FEBRUARY 2018 


TOWARD DATA-DRIVEN 
EDUCATION SYSTEMS 


Insights into using information to 
measure results and manage change 


Samantha Custer 
Elizabeth M. King 
Tamar Manuelyan Atinc 
Lindsay Read 

Tanya Sethi 


enter for 


Bb LT Gniveresl Education y AN Al DDATA 


at BROOKINGS rch Lab at William & Mary 


Samantha Custer is director of policy analysis at AidData 

Elizabeth M. King is a nonresident senior fellow at the Center for Universal Education at Brookings 
Tamar Manuelyan Atinc is a nonresident senior fellow at the Center for Universal Education at Brookings 
Lindsay Read was a research analyst at the Center for Universal Education at Brookings 

Tanya Sethi is a senior research analyst at AidData 


Acknowledgments 


The authors would like to thank Takaaki Masaki, Matthew DiLorenzo, Rebecca Latourell, John Custer, and 
David Batcheck for their invaluable contributions to this report. The authors appreciate the comments 
from peer reviewers who read an early version of the report and thus helped refine our thinking, 
including: Husein Abdul-Hamid, Shaida Badiee, Albert Motivans, and Eric Swanson. 


The Brookings Institution is a nonprofit organization devoted to independent research and policy 
solutions. Its mission is to conduct high-quality, independent research and, based on that research, to 
provide innovative, practical recommendations for policymakers and the public. AidData is a research 
lab at William & Mary that equips policymakers and practitioners with better evidence to improve how 
sustainable development investments are targeted, monitored, and evaluated. Its work crosses sectors 
and disciplines, serving the unique needs of both the policy and academic communities, as well as 
acting as a bridge between the two. AidData uses rigorous methods, cutting-edge tools, and granular 
data to answer the question: who is doing what, where, for whom, and to what effect? 


Brookings gratefully acknowledges the program support provided to the Center for Universal Education 
by the William and Flora Hewlett Foundation. 


Brookings and AidData recognize that the value they provide is in the absolute commitment to quality, 
independence, and impact. Activities supported by their donors reflect this commitment. 


I TOWARD DATA-DRIVEN EDUCATION SYSTEMS 


Table of Contents 

1. Introduction: The role of information in education 1 
From information to impact: A theory of change 2 
2. Do investments in education data match use? 7 
A growing store of data and evidence 7 
Evidence of information use 9 
3. Identifying data needs: What do education decision-makers want? 17 
About the data On i 18 
What is the role of data and information in decisions? 19 
Is the current supply of data meeting the demands of education decision-makers? 25 
How can education data be more useful in decision-making? 27 
4. Conclusion and key findings 32 
References 35 
Appendix A: Composition of survey participants and additional figures 38 
Appendix B: Types of education data 45 
Appendix C: Available datasets on selected education indicators _ : : : : 47 
Appendix D: Survey methodology and questionnaires 49 
Figures, Tables, and Boxes 

Figure 1. Aid to statistics: Trends in volume and as a share of ODA, 2006-15, commitments 3 
Figure 2. Data and evidence: From generation to use and impact 4 
Figure 3. Number of impact evaluations in education, by year 9 
Figure 4. Challenges for education management and information systems a 
Figure 5. Impact evaluations in education, by country 15 
Figure 6. What is the most important factor influencing decisions in various education activities? 2) 
Figure 7. For what purposes do education decision-makers use information? : : : : : 22 
Figure 8. For what purposes do education decision-makers use information, by stakeholder group? 23 
Figure 9. What types of data do education decision-makers use? 24 
Figure 10. How granular is the information education leaders use? 25 
Figure 11. What types of data are most essential for education decision-makers? 26 
Figure 12. What makes some sources of data and analysis more helpful to education decision-makers? 30 
Figure 13. What improvements can make information more helpful to education decision-makers? 31 
Figure 14. Which solutions are the most important to enhance the value of data in education? 31 
Table 1. How different education decision-makers use information? 5 
Table 2. Stated use of EMIS initiatives, by country 10 
Table 3. Areas of decision-making which used national student assessments, by country 13 
Table 4. Types of uses of impact evaluations and systematic reviews (associated with 3ie) 14 
Table 5. Education decisions included in the 2017 Snap Poll, by domain 19 
Table 6. Data needs in the education sector, by decision-making domain 27 
Box 1. Types of large-scale assessments of student learning 8 


Box 2. Comparing the surveys 18 


1. Introduction: The 
role of information in 


education 


Today, 650 million children around the 
globe are at risk of being left behind as they 
fail to learn basic skills. Inequitable access to 
education is part of the problem, but even 
when children are in school, they may not be 
learning. In Uganda, for instance, barely half 
of grade 6 children read at a grade 2 level 
(Uwezo, 2016). In India, just one in four chil- 
dren enrolled in grade 5 can read a simple 
sentence or complete simple division prob- 
lems (ASER Centre, 2017). 


These challenges are widespread. Accord- 
ing to the International Commission on 
Financing Global Education Opportunity 
(Education Commission; 2016), only one in 
ten children in low-income countries (four 
in ten in middle-income countries) are on 
track to gain basic secondary-level skills by 
2030. Moreover, the obstacles to learning 
disproportionately affect marginalized 
populations—children in poor households 
or rural areas (especially girls), children with 
disabilities, and children affected by conflict 
and violence. 


It is clear that the status quo is not good 
enough, but what should be done dif- 
ferently? While struggling schools would 
certainly benefit from better facilities and 
more teachers, research underscores that in- 
put-oriented solutions are likely insufficient. 
Many countries that dedicate substantial re- 
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sources to education still fall short of ensur- 
ing that all children are learning. Meanwhile, 
relatively resource-poor education systems 
in Latvia and Vietnam, for example, punch 
above their weight in achieving greater gains 
for students than their peers with similar in- 
come levels (World Bank, 2018). 


Parents, teachers, policymakers, and school 
administrators need better tools to diagnose 
where and why learning gaps exist, and 
assess what strategies they can employ to 
turn things around. High-quality data and 
evidence are essential for both tasks. 


Numerous governments, organizations, and 
companies have responded to this chal- 
lenge and are generating copious amounts 
of data and analysis to support education 
decision-making around the world. None- 
theless, large gaps remain, as data manage- 
ment processes at the school and national 
level are often under-funded, ad hoc, and of 
variable quality and timeliness. 


While continued investments in data cre- 
ation and management are necessary, the 
ultimate value of information is not in its 
production, but its use. Herein lies one of the 
biggest challenges of translating information 
into actionable insights: those that produce 
education data are often far removed from 


those that make crucial decisions about ed- 
ucation policies, programs, and investments. 
With limited insight on what decision-mak- 
ers use and need, the likelinood of discon- 
nect between supply and demand is high. 


Yet, there has been surprisingly little system- 
atic research on the types of information 
education decision-makers in developing 
countries value most—and why. Much of the 
available evidence on the use of education 
data in developing countries relies upon in- 
dividual case studies. These qualitative snap- 
shots offer deep insights on use patterns and 
challenges in a single context, but make it 
difficult to draw broader conclusions. 


In this report, we offer a unique contribution 
to this body of knowledge by analyzing the 
results of two surveys of education policy- 
makers in low- and middle-income countries 
that asked about their use of data in deci- 
sion-making. Survey participants include 
senior- and mid-level government Officials, 
in-country staff of development partner or- 
ganizations, and domestic civil society lead- 
ers, among others (see Appendix A for more 
information). Respondents do not include 
local-level officials, school administrators, or 
teachers. 


This report aims to help the global education 
community take stock of what information 
decision-makers use to measure results 
and manage change. We define informa- 
tion broadly, including raw statistical and 
administrative data, quantitative and qual- 
itative analysis, learning assessments, and 
the results of program evaluations. Drawing 
upon our review of the literature and the two 
surveys of end users in developing countries, 
we offer practical recommendations to help 
those who fund and produce education 
data to be more responsive to what deci- 
sion-makers want and need. 


In the remainder of this chapter, we articu- 
late our working theory of change that charts 


the path from information generation to use 
(i.e, how education systems transition from 
being data-rich to data-driven). In Chapter 2, 
we synthesize what past studies reveal about 
how data have influenced education policy, 
programs, and practice, paying particular 
attention to the motivations and incentives 
that appear to play a role in both the produc- 
tion and use of education data. In Chapter 3, 
we present the findings from two surveys of 
education stakeholders conducted in 2017, 
with the specific aim of identifying what 
data they use, how data are used, and how 
data can be more useful for policy decisions 
and actions. Chapter 4 concludes with sev- 
eral implications for the future of education 
data investments. 


From information to 
impact: A theory of change 


Data has emerged at the forefront of the 
global develooment agenda. Indeed, the 
United Nations (U.N.) issued a Report of the 
High-Level Panel of Eminent Persons on the 
Post-2015 Development Agenda calling for a 
“data revolution.” 


Recent landmark reports echo this revolu- 
tionary zeal for more and better data in the 
education sector. For example, the Education 
Commission’s Learning Generation report 
argues that “setting clear priorities and high 
standards, collecting reliable performance 
data to track system and student progress, 
and using data to drive accountability are 
consistent features of the world’s most im- 
proved education systems” (2016, p. 52). The 
2016 Global Education Monitoring report 
champions the generation and use of edu- 
cation data, particularly learning metrics, to 
realize the promise of education for all (UN- 
ESCO, 2017). The first World Development 
Report on education, Learning to Realize 
Education's Promise, reiterates the need to 
measure learning to catalyze action: “Lack of 
data on learning means that governments 
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can ignore or obscure the poor quality of 
education, especially for disadvantaged 
groups” (World Bank, 2018, p. 91). 


The international response to the call for a 
data revolution has been positive. ODA sup- 
port for statistics has been increasing over 
the last decade, more than doubling from 
2006 and reaching $541 million in 2015 (Fig- 
ure 1). While this long-term positive trend is 
encouraging, ODA support for data still rep- 
resents only a miniscule 0.3 percent of total 
ODA and only a handful of bilateral agencies 
account for nearly four-fifths of the aid for 
statistics. Moreover, year-to-year fluctuations 
indicate less than predictable support, as 
Figure 1 suggests. 


1 Open Data Watch (2016) reports an impressive rise in global 
investments in statistical capacity between 2015 and 2016— 
from $264 million to $328 million—but, when comparing 
only the donors for which data are available in both years, 
the annual estimated contribution decreased by 10 percent. 
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Ultimately, these investments in data cre- 
ation must be matched by an equal (or 
greater) emphasis on increasing the use of 
evidence by decision-makers to allocate re- 
sources, plan programs, and evaluate results. 
The path from data generation to use, how- 
ever, is not simple, automatic, or quick. The 
seemingly straightforward story of informa- 
tion supply, demand, and use is complicated 
by users’ norms (how they prefer to make 
decisions), relationships (who they know 
and trust), and capacities (their confidence 
and capability to turn data into actionable 
insights). The process of moving from data 
generation to use and ultimately to an im- 
pact on education outcomes must also take 
into account different institutional operating 
environments (i.e., political context) that may 
incentivize or dampen efforts to make deci- 
sions based upon evidence. 


Figure 2 illustrates the complex chain from 
data generation to use and impact. Each 


vdo jo ~%ese 


link in this chain involves different tasks and 
several actors. 


Advances in technology and connectivity 
have largely democratized the generation 
of education data, particularly in front- 
line service delivery contexts such as local 
schools. Multiple levels of government (local, 
provincial, national) are involved in collecting 
data on education inputs and outcomes. 
These officials collect, verify, curate, store, 
analyze, and communicate information. 
Country-specific data are also generated 
and analyzed by others outside government, 
such as researchers in academic institutions, 
non-governmental organizations, interna- 
tional development agencies, and even 
parents and teachers. 


To move from generation to impact, de- 
cision-makers must first take notice of 
available data, interpret it, and link it to the 
roles that they play in the education system 


(Coburn, Honig, and Stein, 2009). Only then 
can they use the data to inform specific de- 
cisions regarding how to allocate resources, 
set policies and standards, or make course 
corrections. 


The ultimate objective of evidence-based 
policymaking in the education sector is to 
fuel progress toward three outcomes: im- 
proved student learning, increased equity, 
and stronger accountability relationships 
among policymakers, schoo! administrators, 
teachers, parents, and students. Scholars 
suggest two avenues through which the use 
of data can lead to these desired outcomes: 
(1) improving the quality of decisions made 
and (2) strengthening the mechanisms 
available to monitor progress and motivate 
responsiveness (See Best et al., 2013; Kel- 
laghan et al. 2009; Jacob, 2017; UNESCO, 
2013; World Bank, 2018). 


» Power relationships 


Institutional Context 
» Role and decision-making capabilities 


» Data culture in bureaucracy & civil society 
» Capacity and resources 
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1. Better decision-making. 

Rather than making decisions based upon 
gut instinct, personal priorities, or anecdotal 
evidence, decision-makers can use disag- 
gregated data to pinpoint problems and 
assess the merits of possible solutions. In this 
respect, use of empirical data and analysis 
can help decision-makers tackle difficult 
questions of how to bolster learning, reduce 
wasteful spending, and target resources ef- 
ficiently to areas of greatest need or highest 
return. Empirical data and analysis can also 
arm policymakers with information they can 
use to counter vested interests and make 
an evidence-based case to mobilize public 
support for difficult reforms. 


2. Stronger monitoring and accountability. 
The regular collection of data and _ infor- 
mation allows for more consistent assess- 
ments of the functioning of the education 
system-—students, teachers, schools, and 
policies—based on objective performance 
indicators and targets. Such assessments 
help all stakeholders, including parents 
and the public, stay up-to-date on how the 
education system is performing (Read and 
Atinc, 2017). These instruments also open up 


opportunities for learning and adaptation by 
school actors and can restore confidence in 
education service delivery as goals are met. 


Unfortunately, not all education data are 
used in these ways. Whether or not policy- 
makers embrace evidence-based practice is 
largely shaped by their conception of what 
is valid evidence, their technical capacity to 
understand available data and analysis, and 
their own “cost-benefit calculus” regarding 
the effort needed to make decisions based 
upon evidence rather than other factors. The 
likelinood that data are effectively used in the 
decision-making process is highly influenced 
by the extent to which data availability is ac- 
companied by an institution-wide culture of 
open communication (or information shar- 
ing), appreciation of data, and accountability 
for results. 


Education systems are complex and multi- 
tiered. Staff have different responsibili- 
ties-from upstream policy formulation to 
downstream classroom instruction—each 
with their own incentive structures and 
data needs. In promoting a culture of evi- 
dence-based decision-making, leaders must 


Ministry of Education (MoE) 


and decentralized units 


The MoE and education sub-min- 


Local Governments 


School Administrators 


istries use education data for 
policy design, strategic planning, 
and decision-making. 


The MoE diagnoses strengths 
and weaknesses of the system, 
measures and ensures equity 
within the system, monitors the 
distribution of resources, and 
holds the system accountable for 
making progress toward defined 
standards and objectives. 


Local planning units use data to 
allocate resources, identify and 

support low-performing schools, 
monitor the implementation of 
education programs, and gener- 
ate comparisons across schools. 


Depending on the level of 
autonomy, some sub-national 
governments are able to plan 
and execute action plans and 
allocate financing based on local 
needs. 
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School leaders use data to track 
progress toward system targets, 
formulate school action plans, 
guide school-level practices, and 
evaluate and support teachers 
and staff. 


ensure that staff at all levels not only have 
access to data that is immediately relevant 
to their needs, but that the information they 
create feeds into the decisions of others (see 
Table }). 


For instance, higher-level administrators 
responsible for student testing and account- 
ability are likely to focus on the reliability of 
tests, whereas local district officials responsi- 
ble for meeting national standards are more 
likely to use those tests to measure achieve- 
ment gaps across schools. Civil society 
groups may use test scores as a barometer 
of how well their neighborhood schools are 
performing, while employers are primarily 
interested how student performance signals 
the quality of the new graduates available to 
them. 


A strategy for data generation and use must 
reflect these differences in the perspectives 
and roles of various stakeholders. There are in- 
stances when local concerns are misaligned 
with national priorities or standards. For ex- 
ample, school-level mechanisms to monitor 
teacher performance may not connect to 
up-stream decisions about compensation 
and in-service training. Similarly, reforms tar- 
geted at strengthening social accountability 
may be limited in cases where public feed- 
back is not integrated into decision-making 
processes (Read and Atinc, 2017). 


This chapter articulated a theory of change 
for how data, in the hands of motivated deci- 
sion-makers, can lead to improved education 
outcomes. In Chapter 2, we turn from the 
theory of how data informs education sector 
decisions to examining what this looks like in 
practice based upon available evidence. 
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2. Do investments 
in education data 


match use? 


The global education conversation has shift- 
ed focus in recent years from raising enroll- 
ments to improving learning outcomes. This 
refocusing of education priorities has spurred 
a notable rise in national and international 
investments to measure student learning. 
However, it is less clear whether these data 
investments are substantively informing the 
design, delivery, and monitoring of educa- 
tion programs. 


In this chapter, we assess the current state 
of investments in data generation, particu- 
larly efforts to strengthen education man- 
agement and information systems (EMIS), 
large-scale student assessments, and impact 
evaluations of policies and programs. We 
also review the existing evidence of whether 
and how education sector decision-makers 
use these information sources. 


A growing store of data and 
evidence 


A learning-focused education system must 
capture accurate, timely, and comparable 
data that link inputs (e.g., school resources 
and financing) to outputs (e.g., school en- 
rollment and attendance) and outcomes 
(eg., performance assessments and other 
quality indicators, see Appendix B: Types of 
education data). Demand for more and bet- 
ter education data has given rise to various 
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frameworks, compacts, and task forces to 
overcome large data gaps. The World Bank 
alone has financed over 200 projects related 
to EMIS in a total of 89 countries between 
1998 and 2013 (Abdul-Hamid et al., 2017). 


Moreover, global and local initiatives have 
emerged to facilitate the production of 
learning data. Consequently, participation 
in international, regional, and citizen-led 
learning assessments has grown over the 
past two decades in low- and middle-in- 
come countries (Box 1). In 2015, for example, 
72 countries participated in the Program for 
International Student Assessment (PISA), up 
from 42 in 2001, with an additional seven 
countries involved in PISA for Development 
(Lockheed, 2015). Similarly, participation in 
the Trends in International Mathematics and 
Science Study (TIMSS) increased from 26 to 
51 countries for the 4th-grade test between 
2003 and 2015 (NCES, 2017). Regional initia- 
tives on student assessments for countries in 
Africa and in Latin America and the Caribbe- 
an have also increased their coverage. 


More countries than ever before are also 
implementing their own large-scale national 
assessments of learning. According to the 
UNESCO Institute of Statistics (UIS) Learn- 
ing Assessment Capacity Index, 127 of 255 
countries (54 percent) conducted a national 
assessment between the years of 2010 and 
2015. These national assessment systems are 


National and sub-national 
learning assessments 


International and regional 
learning assessments 


Citizen-led learning 
assessments 


National and sub-national learning assessments regularly track and assess 
whether students are mastering the national curriculum, in which areas 
students are stronger or weaker, whether certain population groups are 
lagging behind and by how much, and which factors are associated with 
better student achievement. Assessments are census-based or capture 
representative samples of students across countries or provinces. 


Globally and regionally benchmarked assessments allow comparable 
assessments of performance across countries, which can provide a check 
on information that emerges from national assessments. Examples of 
international assessments include the Program for International Student 
Assessment (PISA), Trends in International Mathematics and Science Study 
(TIMSS), and Progress in International Reading Literacy Study (PIRLS). 
Regional assessments include the Southern and Eastern Africa Consortium 
for Monitoring Education Quality (SACMEQ), the Programme for the Analysis 
of Education Systems (PASEC) in francophone West and Central Africa, and 
the Latin American Laboratory for Assessment of the Quality of Education 
(HEEGE) 


Citizen-led assessments measure learning outcomes for children both 
in and out of school. Such assessments, led by civil society organizations 


Source: Adapted from the World Development Report (2018). 


thought to be more relevant than interna- 
tional assessments in designing pedagogical 
reforms and changing the way resources are 
allocated to schools and classrooms (Clarke, 
2012). 


Policymakers and practitioners also have 
access to a larger body of research than ever 
before about the determinants of increased 
learning and equity within education sys- 
tems. For instance, the volume of impact 
evaluations completed has increased three- 
fold since 2005 (Figure 3).2 These studies 
pinpoint reforms, investments, and commu- 


2 These rigorous methods of impact evaluation, experimental 
or quasi-experimental, require that there is a clearly 
identified counterfactual in order to estimate the impact of 
programs or policies. 


such as the ASER Center in India and Uwezo in East Africa, involve parents 
and community stakeholders to yield learning metrics on both access and 
quality of education systems. Citizen-led assessments are of particular im- 

portance in settings where official assessments are of questionable quality. 


nity initiatives that have made a quantifiable 
difference in learning outcomes. In addition, 
within the last five years, a number of critical 
meta-reviews have further clarified what 
works to improve student learning.’ 


There is no question that education informa- 
tion is becoming more abundant—but is it 
being used by those making consequential 
decisions about where to devote scarce 
resources and how to design programs in 
order to maximize student learning? 


3 Murnane and Ganimian (2014); Andrabi, Das, and Khwaja. 
2014; Krishnaratne, White, and Carpenter (2013); Conn (2014); 
Blimpo and Evans (2011); McEwan (2015); Snilstveit, et al. 
(2016); Evans and Popova (2016); Glewwe and Muralidharan 
(2015): Kremer, Brannen, and Glennerster (2013). 
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Source: 3ie Impact Evaluation Repository. www.3ieimpact.org/en/evidence/impact-evaluations 


Collecting, processing, and communicating 
data requires substantial resources. There- 
fore, it is imperative that we assess whether 
data produced are indeed accessible and 
valuable to key decision-makers. The next 
section examines what the existing evidence 
can tell us about the current state of use of 
data in education decisions. 


Evidence of information use 


When data advocates promote evi- 
dence-based decisions in education systems, 
they rarely specify who are the intended 
users, for what purpose, and what kinds of 
data are needed. The implicit assumption 
is: by everyone, for everything, and any data. 
However, the reality is more sobering. There 
is little indication that decision-makers are 
systematically using education data and 
analysis to inform their policies or decisions. 


There are many potential effects of evidence 
on stakeholders—some intrinsic (e.g., a great- 
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er appreciation of the role or state of edu- 
cation among society writ large) and some 
instrumental (e.g., improving specific edu- 
cation outcomes through motivating better 
decisions).* Given the difficulty of assessing 
the more intrinsic or symbolic role of infor- 
mation, here we employ an instrumentalist 
approach, restricting our focus on the use of 
data to inform specific policies. 


EMIS 

An Education Management Information 
System (EMIS) produces data that could be 
of tremendous value for the design, imple- 
mentation, and monitoring of education 
programs. Table 2 summarizes the stated 
uses of ten EMIS initiatives, including facili- 
tating information exchange between levels 


4 Boswell (2014) distinguishes between two uses of infor- 
mation: legitimizing and substantiating. Policymakers can 
use information to bolster their credibility in taking sound, 
rational decisions, or deploy information to back up claims 
or preferences. For more information, see https://christinabo- 
swell.wordpress.com/2014/06/23/why-real-policy-impact-is- 
more-difficult-to-evidence-than-symbolic-knowledge-use/. 


of government, planning and budgeting, 
performance monitoring, and reporting. 


The EMIS centrally organizes information 
from multiple levels of the education sys- 
tem, collecting and managing critical data 
points such as student enrollments, number 
of teachers, and class size (see Appendix B 
for more information on type of education 
data). Schools or local governments usually 


Information 
sharing 
between 
government 
levels & 
agencies 


Integrating 
hard-to-reach 
areas 


Country 


Bosnia & 
Herzegovina 


Cambodia 


Eritrea 


Guatemala 


Honduras 


Lebanon 
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Lithuania 


Malaysia J 


Nigeria J 


Philippines 


Source: Abdul-Hamid, Saraogi and Mintz (2017) 


Budgeting 
activities 


report these data on a periodic basis, using 
standard forms and guidelines from the cen- 
tral education ministry. 


Countries are increasingly adopting web- 
based dissemination of EMIS data, which is 
making education systems more open and 
transparent. However, the utility of this infor- 
mation is only as good as the strength of the 
underlying data. 


Identifying 
infrastructure 
needs 


Performance 
monitoring 


School 
management 


Reporting on 
system 


v 
v 
v v 
v 
v 
v 
v v 
v v 
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Unfortunately, in many countries, the EMIS 
is not fully functional, which inhibits effec- 
tive monitoring of education policies and 
programs (Abdul-Hamid, Saraogi and Mintz, 
2017). In particular, education administrators 
must tackle several challenges ranging from 
data quality to leadership and capacity, be- 
fore these EMIS are “fit-for-purpose” (Figure 
4). 


Lack of data utilization for 
decision-making 


Data F ‘ 

Untimely production and 

Challenges dissemination of data 

Lack of reliable and quality data 

MIS not functional due to 

System technical problems 
Challenges 

System capacity issues 

Lack of training for data usage 

Coordination issues 

Operational Funding issues 
Challenges 

Long-term sustainability 

MIS not implemented 

Implementation delays 

Leadership changes 
Leadership 

Challenges Lack of data culture 


Lack of clear vision and support 


Anumber of case studies illustrate these chal- 
lenges in practice. A study of EMIS capacity 
in the Philippines, Ghana, and Mozambique, 
for example, concludes that the countries 
were not generating adequate information 
to monitor learning outcomes, inequality, 
and _ cost-effectiveness (DeStefano, 201)). 
Ghana, often cited as a regional leader for its 
data capabilities, still faces major constraints 
of duplicative data systems, limited quality 


NUMBER OF EMIS ACTIVITES 


Note: MIS = Management Information System 
Source: Abdul-Hamid, Saraogi and Mintz (2017) 
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assurance procedures, and over-reliance on 
paper-based and manual data entry pro- 
cesses (Spratt et al., 2011). 


In India, a survey of frontline bureaucrats 
reveals that ambiguity about the purpose 
of school monitoring data can severely un- 
dercut the usefulness of information being 
collected (Bhatty, 2016). “Information [on 
school quality] that is so assiduously collect- 
ed through ever-more sophisticated formats 
is neither analyzed nor followed-up [because 
it is poorly defined]. Instead, the files remain 
locked up in school, block, or district cup- 
boards’ (11). 


LEARNING ASSESSMENTS 

Sponsors of student learning assessments 
emphasize their value for policymaking and 
agenda setting, but the evidence of such use 
is scarce and uneven. Open data on student 
performance has been a boon to researchers 
seeking to explain differences in education 
outcomes at regional and international lev- 
els. However, the link between assessment 
results and educational reforms in countries 
is tenuous at best (Kellaghan, 2009). 


International assessments appear to have the 
greatest visibility in higher income countries. 
Countries like Germany and Norway have 
responded to the release of their PISA re- 
sults with a revision of curriculum standards 
(Breakspear, 2012) and the introduction of a 
national quality assessment system (Baird et 
al., 2011), respectively. 


The challenge of translating awareness of in- 
ternational assessments into action is more 
acute in middle- and low-income countries. 
In Colombia, Indonesia, Jordan, and Turkey, 
the release of PISA and TIMSS results have 
been associated with a subsequent uptick 
in discussion of education reform (Lockheed, 
2015). However, it is unclear whether this 
heightened awareness has provoked action. 


Similarly, Early Grade Reading Assessment 
(EGRA) results triggered debate about the 
quality of education in Senegal, but little 
change in policy (Mejia, 2011). Also similarly, 
citizen-led learning assessments in India, 
Pakistan, and a few African countries have in- 
creased public awareness of poorly perform- 
ing schools, but have not spurred concrete 
action to improve learning (R4D, 2015). 


Comparatively, national learning assess- 
ments have had more traction with policy- 
makers in developing countries. In Jordan 
and Uruguay, the national assessment results 
were used to help teachers improve their 
teaching (Obeidat and Dawani, 2014; Ravela, 
2005). In Bhutan, national assessment results 
spurred a revision of the mathematics sylla- 
bus (Kellaghan et al., 2009). In Kenya, they 
triggered an effort to ensure that all schools 
had sufficient desks and textbooks (ibid). 


Table 3 summarizes evidence cited in re- 
ports about the use of national assessment 
data in decision-making. These data appear 
primarily to support countries’ monitoring 
and evaluation systems, curriculum reform, 
and allocation of school inputs, though the 
intensity of use is not known. They are less 
often utilized in budgeting and to shape 
teacher policies. 


PROGRAM EVALUATION AND 

OTHER RESEARCH 

While school-level administrative data from 
a country’s EMIS and student learning as- 
sessments can support real-time monitoring, 
program evaluations and other research may 
be more helpful in assessing what is and is 
not working and why. 


It makes sense that the growing number of 
program evaluations in the education sector 
would prompt decision-makers to mine 
these data for lessons to inform program 
selection, design, and implementation. In 
fact, many organizations make their evalua- 
tions accessible on their websites and easily 
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Professional Teacher 
Curriculum | Supportto | Student Staff develop- compensa- School | Strategy / 
Country reform teachers support | deployment ment tion Budgeting | _ inputs M&E 


Bhutan WA 
Bolivia J 


Burkina 
Faso v 


Ethiopia WA 


Gambia, JV JV Jv 


The 


Guinea Jv 


India Jf A WA 
Jordan Jv J 
Kenya WA 


Nepal v 


Sri Lanka v 
Uganda Jv 
Uruguay mw 


Vietnam JV v v 


Zimbabwe J mY WA 


Source: Authors’ analysis based on information provided in Abdul-Hamid, Abu-Lebdeh, and Patrinos (2011), Kellaghan, Greaney, and Murray (2009); Obeidat 
and Dawani (2014); Ravela (2005); Senghor (2014), Tobin, et al. (2015), UNESCO (2013). 
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digestible through short policy briefs for this 
purpose. In addition to individual program 
evaluations, they also provide meta-reviews 
that sift through hundreds or thousands of 
published studies to distill relevant insights. 


Table 4 draws upon data from one of these 
organizations—the International Initiative for 
Impact Evaluation (3ie)—which tracks how 
their evaluation studies are being used by 
decision-makers. While not limited to the 
education sector, the data shows that impact 
evaluations are influencing decision-making 
across sectors, particularly with regard to 
informing course corrections, as well as dis- 
cussion and design of policies and programs. 


Nonetheless, two reviews by the Global Part- 
nership for Education (GPE) suggest that 
research and analysis have limited influence 
on the education sector plans of the coun- 
tries it supports. According to Bernard (2015), 
only 18 of 42 country plans (43 percent) used 
an education sector analysis to inform their 
policies and even fewer cited rigorous anal- 
ysis to identify root causes of performance 


Change policy or program design AIS 
Inform discussion of policies & programs 25.8 
Inform design of other programs Z5u/ 
Inform global policy discussions 13 
Take successful programs to scale 8.2 
Close programs that do not work Sal 
Improve the culture of evaluation use & (0) 


strengthen the enabling environment 


Total 100 


Source: Executive Director's Report to the Eighteenth Meeting of the 3ie Board 
of Commissioners, London, November 7, 2017, page 9. 


challenges or determine sector priorities, 
even when the information was readily avail- 
able. 


Similarly, an earlier assessment of 46 sector 
plans and joint sector reviews found that 
these documents rarely discussed learning 
outcomes or cited empirical evidence (e.g., 
education production functions, random- 
ized trials, meta-analyses, surveys) in articu- 
lating their approach to improving learning 
outcomes (GPE, 2012). While education 
sector plans are only one possible outlet, 
among many, for decision-makers to make 
use of available program evaluations or other 
research, this apparent disconnect between 
evaluations and forward-looking planning 
warrants further scrutiny. 


While we will probe this question in greater 
depth in the next chapter using the survey 
results, here we will make three observa- 
tions. First, education decision-makers will 
only use evaluation data that is relevant to 
them. This might be easier said than done. 
The 3ie repository is a case in point: over 
half of the 855 impact evaluations in the 
education-sector pertain to just 10 countries 
(Figure 5). Education stakeholders interested 
in other geographic areas are out of luck. 


Second, evaluation studies tend to be 
funder- and researcher-driven, and thus may 
not cover the specific programs or topics of 
interest to a broader set of education stake- 
holders. 


Finally, published evaluation studies typically 
focus on programs that have shown some 
impact, but education decision-makers are 
interested in learning from not only program 
successes, but also their failures to avoid 
common pitfalls. 


Beyond evaluation studies that focus on spe- 
cific policies or programs, other initiatives, 
such as the Research on Improving Systems 
of Education (RISE) program and the World 
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Bank's Systems Approach to Better Educa- 
tion Results (SABER), aim to produce a rich 
body of analytical work that could improve 
our understanding of the underpinnings of 
progress in education outcomes in develop- 
ing countries. 


100% 


SABER, in particular, offers system-level 
diagnostics on the state of education in de- 
veloping countries. The diagnostic toolkit en- 
ables educators and policymakers to assess 
education policies and practices in light of ee 
global standards and best practices. 
There is some indication that SABER diag- Kenya 
nostics are being used to influence country 
reforms and dialogue with development Chile 
agencies.° For example, one case study doc- 
uments steps that Jordan has taken (based Brazil 
upon background information from SABER) Colombia 
to strengthen its student assessment sys- South Africa 
tems through linking student assessments 
with teacher training and support, as well as 
disseminating assessment results (Obeidat 
and Dawani, 2014). 


Peru 


50% Pakistan 


0% 


Source: 3ie Impact Evaluation Repository. 
www.3ieimpact.org/en/evidence/impact-evaluations/ 


5 See country briefs on the program website: 6 China (81), India (68), Mexico (62), Kenya (61), Chile (33), Brazil 
http://saber.worldbank.org/index.cfm?indx=6&sub=5. (32), Colombia (29), South Africa (28), Peru (26), Pakistan/ 
Uganda (both, 24). 
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In this chapter, we distilled insights from the 
available literature to understand how deci- 
sion-makers use three sources of evidence— 
data from a country’s EMIS, student learning 
assessments, and program evaluations—to 
inform education policy and practice. The 
scorecard is mixed at best: investments in 
data generation appear to be outstripping 
the use of this information to strengthen 
education systems, aside from a few bright 
spot examples. 


However, there is still much that we do not 
know about the data decision-makers use 
and the evidence that they want to be more 
effective in their jobs. In Chapter 3, we ana- 
lyze the results of two recent surveys of edu- 
cation decision-makers in order to shed light 
on these questions from the perspective of 
one important user base: national-level deci- 
sion-makers and influencers involved in set- 
ting and informing education policy across 
public, private, and civil society spheres in 
low- and middle-income countries world- 
wide. 
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3. Identifying data 
needs: What do 
education decision- 
makers want? 


To move from data generation to policy 
impact, we need better intelligence on the 
barriers to evidence use and the types of 
information that decision-makers want. In 
the previous chapter, we examined available 
evidence of education data investments and 
use, but were left with more questions than 
answers. 


In 2017, AidData fielded two surveys of na- 
tional-level policymakers and practitioners 
in low- and middle-income countries who 
shared their experiences of how they source 


and use data in their work, as well as what 
would be most helpful to them in the fu- 
ture. In this chapter, we analyze these novel 
datasets to answer three key questions: what 
data are in demand, by whom, and why? 


We will discuss these findings in more detail 
in the remainder of this chapter. The follow- 
ing section first provides more background 
on the survey data to set the findings in 
context. 
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Listening to Leaders Survey. The 2017 Listening to Leaders Survey was sent via email to 
policymakers and practitioners knowledgeable about, or directly involved in, development policy 
initiatives at any point between 2010 and 2015, in 126 low- and middle-income countries. We 
were successfully able to send a survey invitation to roughly 47,000 of the 58,000 individuals 
who met our inclusion criteria (about 80 percent). Of the 47,000 people who received an 
invitation, 3,500 (7.4 percent) participated in the survey. 


Education Snap Poll. We identified a subset of the broader Listening to Leaders sampling frame 
of individuals who held positions of relevance to the education sector to use as the sampling 
frame for the 2077 Education Snap Poll. Of the 2,000 individuals who were sent the survey 
invitation, 180 leaders (9 percent) responded. 


Both surveys captured the views of five stakeholder groups. The profile of education sector 
respondents of both data sources is broadly similar: around 40 percent are host government 
officials and most are from sub-Saharan Africa. Compared to the snap poll, the share of in- 
country development partner respondents was smaller in the 2017 Listening to Leaders Survey; 
and the corresponding share of CSO/NGO respondents was larger. 


For more information on the composition of the survey respondents please see Table Al and A2 
in Appendix A. 


About the data to examine the various roles that education 


stakeholders perform and their specific data 
AidData’s 2017 Listening to Leaders (LtL) Sur- needs: 
vey captured the views of nearly 3,500 survey 
participants in 126 low- and middle-income 
countries from 22 policy domains, including 
education. Their insights shed light on the 
broader picture of data and evidence use, as 
well as how the education sector is different 
from other social sectors.” 


The respondents of the two surveys included 
representatives from five stakeholder groups: 
government officials, development partner 
organizations, civil society groups and NGOs, 
the private sector, and independent experts. 


Given the relatively small sample size for the 
education poll, we primarily draw insights 
regarding the survey respondents overall, 
though in some cases we mention differ- 
ences among stakeholder groups. The ma- 
jority of the education poll respondents have 
roles that support policymakers who make 
decisions related to these domains. Some 


7 The health and education sectors, for example, have a similar also make final decisions related to various 
service delivery model, but their investment strategies hint at education activities. 
very different views on the value of data. Only three percent 
of official development assistance for education is spent on 
global public goods such as data and research, compared 
with 20 percent for health (Schaferhoff and Burnett, 2016). 


Specific to this study, AidData and the Brook- 
ings Institution fielded a more targeted 
survey of decision-makers in 126 countries. 
Approximately 180 leaders from '78 countries 
responded to the 2077 Education Snap 
Poll. The poll provides a unique opportunity 
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In interpreting the results, it is important to 
recognize that the focus in both surveys is 
very much on national-level decision-mak- 
ers. As such, these data give insight into 
what some user groups care about, but not 
the needs and concerns of other groups (e€.g., 
parents, teachers, school-level administra- 
tors, local officials). 


What is the role of data and 
information in decisions? 


Rather than studying data use in the ab- 
stract, survey respondents offered their 
insights on the types of decisions they make 
in the education sector and the role infor- 
mation plays—among other factors—in that 
process. Government officials, for example, 
may allocate resources, determine quality 
standards, and hold school administrators 
accountable for meeting national targets.® 
Civil society leaders may advocate for more 
effective government-run schools or directly 
administer their own programs that are sub- 
ject to national standards. 


Using their survey responses, we can gain 
insight into the extent to which education 
data or analysis is a driver of these decisions 
in practice. Specifically, we asked partici- 
pants in the 2017 Education Snap Poll about 
the role of information versus other factors 
in driving ten common education decisions, 
adapted from the OECD's Education Sector 
at a Glance (2012). We further categorized 
these decisions into four decision-making 
domains: (1) organization of instruction; (2) 
personnel management; (3) resource man- 
agement; and (4) planning and structures 
(see Table 5). 


8 School administrators supervise teachers, implement school 
budgets, and report on student enrollment and progression. 
It should be noted that while these front-line implementers 
are an important group of education data users, our survey 
results are primarily capturing use patterns of national-level 
leaders. 
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Personnel 
management 


Hiring and deploying 
teachers or principals 


Developing careers and 
assessing performance of 
teachers and/or principals 


Determining 
compensation for 
teachers/principals 


Planning and 
structures 


Designing and defining 
programs of study and 
course content 


Creating or closing/ 
abolishing schools or 
grades 


Planning and developing 
strategies 


Source: Decision-making domains adapted from OECD Education 
at a Glance (2012) 


Leveraging responses from the 2017 Listening 
to Leaders Survey, we can also pinpoint the 
primary purposes for which decision-makers 
and influencers use education data or anal- 
ysis in their work. Survey respondents could 
identify several possible use cases, including: 
program design, program implementation, 
advocacy and agenda-setting, capacity 


building and technical assistance, monitor- 
ing and evaluation, research and analysis, or 
external communications. 


In the remainder of this section, we analyze 
how decision-makers responded to these 
two questions on the role of information writ 
large and the particular purposes for which 
they use education data or analysis. 


Finding 1: Having enough 
information is seldom the decisive 
factor in making most education 
decisions; instead, decision- 
makers point to having sufficient 
government capacity. 


In determining whether to enact education 
policies, modify programs or allocate resourc- 
es, decision-makers assess the anticipated 
benefits and costs of the options before 
them. In the process, they weigh many fac- 
tors of which data is only one consideration. 


In the 2017 Education Snap Poll, we asked 
policymakers and practitioners to identify 
the most important factor in making or influ- 
encing ten common education decisions.® 
Survey participants identified sufficient 
government capacity to implement [policy 
or programmatic] changes as the decisive 
factor for most decisions in the education 
sector. 


So, where does data or analysis fit into the 
decision-making process? In Figure 6, we see 
that leaders view having sufficient informa- 
tion as less consequential in how decisions 
are made than technical capacity, financing, 


9 Respondents first selected all the activities they were person- 
ally involved with and then identified the most important 
factor in making or influencing those decisions. See Figure 
6 for the response options. While respondents could have 
interpreted “sufficient government capacity” in a number of 
ways, we think it reasonable to interpret this as capacity to 
implement programs or policies. See Appendix D for the full 
Education Snap Poll questionnaire. 


and political support. It should be noted that 
this does not necessarily mean that these 
decision-makers view this as the ideal situa- 
tion, merely the status quo. That said, these 
results are consistent with prior studies, such 
as that by Bruns and Schneider (2016) which 
show that political considerations have 
stymied education reforms in several Latin 
American countries, even when empirical 
evidence justifies reforms. 


There could plausibly be a mutually rein- 
forcing relationship between government 
capacity and the perceived importance of 
information in decision-making. Shortage 
of staff in national statistical organizations 
and ministries and limited capacity for using 
and analyzing data have been reported to be 
among the most critical constraints to data 
use in Honduras, Timor-Leste and Senegal 
(Custer & Sethi, 2017). 


Is the availability of sufficient information 
more critical to some decisions than others? 
Leaders place a somewhat higher premi- 
um on having enough information when 
it comes to decisions such as creating or 
abolishing schools or grades, and testing, 
assessing or credentialing students (Figure 
6).'° One possible explanation for this could 
be that leaders feel that they need stronger 
justification (via an evidence base) for these 
decisions as they could become easily po- 
liticized. Teachers or parents may strongly 
disagree with a school closure, for example, 
and mobilize dissenting voices." 


10 This result is based on a rather small number of responses, 
and therefore should be interpreted with caution, especially 
in generalizing the findings to the education sector as a 
whole. 


In addition to the pre-defined activities, respondents could 
select “other” and write-in responses. These were mapped 
to the decision domains as much as possible, but we 
combined the 15 responses that could not be mapped and 
created a fifth category “other”. Several of these responses 
relate to working with education data, statistics, or analysis. 
It is unsurprising that information (along with public and 
political support) is the most important factor for this group 
of respondents. 


1 


=a 


3. IDENTIFYING DATA NEEDS: WHAT DO EDUCATION DECISION-MAKERS WANT? 20 


Designing student 
activities [27] 
Organization of 


EnStMMCEION Testing or credentialing 
students [20] 
Hiring teachers or 
principals [11] 
Eefsonnel " Assessing teacher 
An performance [17] 
Determining compensation 
for teachers/principals [11] 
Budgeting for education 
resources [41] 
Resource 
management 


Ensuring provision of 
school inputs [17] 
Designing programs and 
content [31] 


Planning and 
structures 


Creating or abolishing 
schools/grades [10] 


Planning and developing 
strategies [74] 


Enough Sufficient Public Sufficient 
information financial and political government factor 
resources support capacity 


Notes: Of the ten activities listed on the left side of this figure, each respondent first selected the activities that s/he was involved in, and then the most 
important factor influencing the decisions pertaining to each activity selected. For each activity, the distribution of responses is visualized from left to right. 
The total number of responses for each activity is noted in parentheses. 


Source: 2017 Education Snap Poll 


Finding 2: Education decision- 
makers employ evidence ina 
supporting role throughout the 
policymaking process, for both 
retrospective assessment and 
forward-looking activities. 
Information may not be the decisive factor 
in education decisions, but it is still one of 


the variables that decision-makers consider. 
When analyzing the results of the Listening 
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to Leaders Survey across all sectors, Masaki et 
al. (2017) observed that, on average, “leaders 
use evidence more to conduct retrospective 
assessments of past performance than to 
inform future policy and programs.” 


Notably, however, decision-makers in the 
education sector are more likely to use data 
and analysis for forward-looking purposes, 
such as design and implementation of 
policies or programs, compared to the use 


Another 


All sectors 


m 
a 
c 
re) 
o 
ad 
° 
3 


Monitoring and evaluation 


Design 


Research and analysis 


Implementation 


Advocacy and agenda-setting 


Capacity building/TA 


External communications 


o 
Re) 
& 


77% 
72% 


76% 
63% 


76% 
73% 


| 
& 


65% 


| 
& 


N 
& 


62% 


51% 
43% 


Note: This figure shows the percentage of respondents in (a) the education sector and (b) all sectors who use evidence for different purposes (n=99 and 
n=1769, respectively). Note that the percentages do not add up to 100 because respondents were able to select all applicable response options. 


Source: 2017 Listening to Leaders Survey. 


patterns observed overall for all sectors. As 
shown in Figure 7, the majority of education 
sector decision-makers (over 7O percent) 
that report using data or analysis, do so fairly 
consistently throughout the policymaking 
process.” This appears to reinforce the earlier 
finding that evidence can play a supporting 
role, even when it is not the major driver of 
education decisions. 


12 The high incidence of use throughout the policymaking 
process is also visible in the health sector. In contrast, the 
governance sector used data primarily for two purposes: M&E 
(77 percent) and research and analysis (73 percent). Figures 
available upon request. 


Nonetheless, education decision-makers are 
not necessarily monolithic, and we see some 
important distinctions between stakeholder 
groups in how they report using information 
in their work. Interestingly, considering their 
oversight of vast public-sector education pro- 
grams, government officials were less likely 
than other stakeholder groups to use data 
and analysis for program implementation or 
monitoring and evaluation (see Figure 8). This 
finding may partly reflect the composition 
of the survey, which includes national-level 
officials, rather than local government repre- 
sentatives or school administrators. 
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[ Government stakeholders [N=39] Non-government stakeholders [N=60] 


Research and analysis 81% Monitoring and evaluation 81% 

Design 77% Implementation 79% 

Advocacy and agenda-setting 74% Design 76% 
Monitoring and evaluation 71% Capacity building/TA 73% 
Capacity building/TA 68% Research and analysis 72% 
Implementation Advocacy and agenda-setting 69% 


External communications 


External communications 43% 


PERCENTAGE OF RESPONDENTS 


Note: This figure disaggregates the results of Figure 7 into two cohorts: government officials and other stakeholder groups in the education sector. 


Source: 2017 Listening to Leaders Survey. 


Is the current supply of data 
meeting the demands of 
education decision-makers? 


As discussed in chapter 2, there is a growing 
wealth of education data, from the prolifera- 
tion of learning assessments, program evalu- 
ations, and the strengthening of country ed- 
ucation Management information systems. 
While data and analysis may not be the 
decisive factor in most education decisions, 
leaders do value and use this information at 
various stages of the policymaking process. 


In this section, we assess whether the cur- 
rent supply of education data is aligned 
with what decision-makers demand. In 
other words, what types of data or analysis 
do education sector decision-makers prize 
most and where are the gaps between what 
they want and the information that is readily 
available to them. 
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Finding 3: Education decision- 
makers most often use national 
statistics from domestic sources 
and program evaluation data from 
international sources for their work. 


Which types of information do education 
sector policymakers and practitioners use in 
their work—and from which sources? Analyz- 
ing their responses to the 2017 Listening to 
Leaders Survey, we find that decision-makers 
overwhelmingly relied on national statistics, 
among domestic sources of information. Of 
the information produced by international 
organizations, education stakeholders were 
most likely to use program or project evalua- 
tion data (Figure 9)." 


Nearly 90 percent of education deci- 
sion-makers reported using national statis- 
tics to support their work, compared with 63 
percent who use program evaluation data or 


13 See Appendix C: Available datasets on selected education 
indicators. 


a Domestic sources 


National statistics 88% 


Survey data 


Program/project 
evaluation data 


Public opinion data 


Government budget/ 
expenditure data 


Spatial or satellite data 


Aid finance data 


International sources 


Program/project 68% 
evaluation data 
National statistics 40% 
Survey data 59% 


Government budget/ 


expenditure data 36% 
Public opinion data 34% 
Aid finance data 28% 


Spatial or satellite data 4% 


PERCENTAGE OF RESPONDENTS 


Notes: This figure shows the percentage of education respondents who use each type of data. 32 (or 35) respondents answered questions about the types 


of data produced by domestic (or international) sources. 
Source: 2017 Listening to Leaders Survey 


survey data (Figure 9, left chart). The outsized 
use of national statistics among education 
decision-makers could reflect the sector's 
reliance on routine administrative data, such 
as enrollments and school infrastructure 
available through the country’s EMIS.“ 


Comparatively, resoondents in the health 
sector reported usage rates for program 
evaluation data (85 percent) and survey data 
(80 percent), that were similar to their use 
of national statistics (84 percent). Similarly, 
decision-makers in the governance sector re- 
ported using survey data and national statis- 
tics in roughly equal measure (75 percent).'® 


More than 80 percent of education respon- 
dents report using both quantitative and 


14 The high use of national statistics points to the salience of 
data for the country in question. This may include statistics 
such as dropout rates for primary school students by district 
or municipality, the number of schools with secondary 
education in each village, or pupil-teacher ratios in urban vs. 
rural areas. The information itself may be at any administra- 
tive level, but pertains to the state of affairs for the country in 
question. 


15 Figures available upon request. 


qualitative analyses, regardless of whether 
domestic or foreign sources produced them. 
Use of impact evaluations does not fall far 
behind, with 75 percent of education sector 
leaders using this information in their work. 


We observe similar use patterns in the 
health and governance sectors, which may 
point to a growing appreciation for using 
mixed-methods approaches to understand 
drivers of progress. These findings also ap- 
pear to affirm what we learned from users of 
3ie impact evaluations in Chapter 2 (Table 3) 
who reported using this information to influ- 
ence policies and programs. Nonetheless, it 
is important to recognize that the term “use” 
does not distinguish between employing 
evidence to justify an existing policy or pro- 
gram, inform a future decision, or assess the 
merits of a past choice. 


Since world leaders adopted the World Dec- 
laration on Education for All in 1990, there 
has been a strong recognition that leaving 
no one behind means shining a light on in- 
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equalities not only between, but also within 
countries. So, how does this affect the use 
of data by education sector policymakers? 
When it comes to domestically produced 
data, most leaders use information disag- 
gregated at the national- (96 percent) or 
sub-national (72 percent) levels (see Figure 
10).°© Since our respondents are primarily 
national-level leaders based in capital cities, 
the use of sub-national data is likely less pro- 
nounced than it would be among local-level 
leaders.”” 


|| Domestic sources International sources 


National 96% close 69% 
national 
Sub- ‘ 
national 72% National 63% 
Cross- Sub- 
national 37% national 47% 


PERCENTAGE OF RESPONDENTS 


Note: This figure shows the percentage of education respondents 
who used information at varying levels of geographical granularity. 
37 (or 45) respondents answered questions on the granularity of 
domestic (or international) information. 

Source: 2017 Listening to Leaders Survey 


Figure 10 highlights an important role for 
international sources of education data, 
namely the provision of cross-national data. 
Countries increasingly benchmark them- 
selves against their neighbors or best practice 
examples in the relevant sector. International 
data sources that report based on common 
standards that allow for cross-country com- 
parisons provide valuable information to 
policy-makers and the public alike. 


16 A similar use pattern is found in the health and governance 
sectors. Within the sub-national category, respondents 
mostly used provincial data (65 percent) and district-level 
information (44 percent). 

17 It is very likely that the type of data used by local-level lead- 
ers and staff is quite different, with possibly a much stronger 
focus on highly granular data that helps them understand 
their area (ward, village or municipality). 
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Finding 4: Education decision- 
makers consider administrative 
data and program evaluations 
most essential, and want more of 
the latter, signaling a gap between 
need and supply. 


To what extent does the data that leaders 
want vary depending upon the nature of 
their work? Analyzing responses to the 2017 
Education Snap Poll, we assess which types 
of data decision-makers deemed most es- 
sential in each of the ten common education 
activity areas we examined previously. We 
also asked survey respondents about their 
wish list - what types of information would 
they want more of? 


Figure 11 provides a breakdown of how de- 
cision-makers rated the most (and least) es- 
sential data types by education activities. De- 
cision-makers responsible for allocating and 
managing resources place a premium on 
administrative data (e.g., number of schools, 
teachers, students) and government bud- 
get and expenditure data (e.g., school-level 
budgets, expenditure per student). For 
those working on personnel management, 
teacher performance data are most valuable 
to hire and compensate staff. Meanwhile, 
leaders tasked with overseeing instructional 
matters deem program evaluation data and 
student-level assessment data essential. 


There were several categories of data that 
decision-makers wished were more readily 
available to support their work (see Figure A5 
in Appendix A)."® In Table 6, we juxtaposed 
these responses about what data leaders 
wished for with the data they deemed most 
essential for their work by each of the four 
decision-domains. Data types that were both 
deemed as essential to leaders’ work and 
also high on their wish lists represent attrac- 
tive investment opportunities for data pro- 


18 Note: 18 percent reported having access to the information 
they need. 


Planning and 
structures [150] 


Organization of 
instruction [61] 


cere ee ee 
ll — 


Resource 
management [45] 


Personnel 
management [18] 


Education Student- School- Teacher Survey Citizen Program Government Aid/ 
admin. level level performance data opinion performance/ budget/ philanthropic 
data assessment data data data evaluation expenditure finance 
data data and data data 


analysis 


Note: The number of responses in each decision-domain are reported in parentheses. For each decision-domain, the distribution of responses among 
various types of data is visualized from left to right. 
Source: 2017 Education Snap Poll. 


ducers to increase their impact in response 
to user demand. We identified four such 
opportunities: (1) program performance and 
evaluation data; (2) budget and expenditure 
data; (3) student-level assessment data; and 
(4) teacher performance data. 


The desire for more evaluations is striking in 
relation to the earlier supply-side discussion 
in Chapter 2. While there has been a steady 
uptick in the last two decades, gaps probably 
remain in the geographic diversity of the ex- 
isting studies and in sharing and disseminat- 
ing the findings of existing evaluations with 
decision-makers in low- and middle-income 
countries. Leaders see this valuable data as 


being in short supply. CSOs report that the 
results of programs that have not worked are 
not made public due to reputational risk, 
hindering future learning from such failures 
(Custer and Sethi, 2017). Meanwhile, the lim- 
ited ability of policymakers to interpret eval- 
uation data is also a serious barrier in using 
research to inform policy (Callen et al., 2017) 


Silos and fragmentation may be a common 
theme across the three remaining data 
investment opportunities. In Chapter 2, we 
foreshadowed that the use of student learn- 
ing assessments was likely hampered by the 
lack of interoperability. Host government 
officials reportedly wish for greater access 
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Essential but not 
high on wish list 
(Met need) 


Decision domain 


Organization of - 
instruction 


Essential and 
high on wish list 
(Unmet need) 


Student-level 
assessment data 


Not essential but 
on wish list 


Aid and 
philanthropic 
finance data; 
survey data 


School level data 


Personnel - 
management 


to education administrative data; however, 
since the government itself often collects 
such data, this suggests that the root issue 
may be lack of access, rather than availability. 
Government ministries are often reluctant to 
share information and instead retain com- 
peting, proprietary systems. These access is- 
sues are compounded for data users outside 
of the government that seek greater access 
to teacher performance data (prioritized by 
CSO leaders) and government budget and 
expenditure data (prioritized by develop- 
ment partners). 


It is worth noting that citizen opinion data, 
while not necessarily deemed as essential, 
is another category of data that appears to 
be in relatively short supply relative to de- 
mand. While only 12 percent of respondents 
consider such data essential to their work, 
26 percent wish more of such data existed 
(see Figures A3 and A4 in Appendix A). One 
possible explanation of this high interest is 
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Teacher 
performance data 


Education 
administrative data 


Aid and/or 
philanthropic 
finance data 


that policymakers may value citizen opinion 
data as a barometer of political support for 
education reforms. 


How can education data 

be more useful in decision- 
making? 

Having identified some of the gaps that 
exist in meeting the needs of education 
decision-makers, we now turn to what pro- 
ducers and funders of data should do better 
or differently to meet the data demands of 
the sector. What information attributes do 


decision-makers want? What improvements 
can make data more usable? 


Neither essential 
nor on wish list 


Finding 5: Education decision- 
makers value domestic data that 
reflect local context and point 

to policy actions; and improving 
the timeliness and accessibility 
of information will make it more 
helpful. 


In the 2017 Listening to Leaders Survey, we 
asked respondents about the attributes of 
information that make it directly helpful for 
their work, broken down by source of infor- 
mation (Figure 12). Data from both domestic 
sources and_ international sources were 
deemed most helpful when they provide 
information that reflects the local context. 
Additionally, education decision-makers 
rated information from international sources 
as most helpful because it provides policy 
recommendations (43 percent) likely in- 
formed by cross-national experience and is 
accompanied by critical financial, material, 
or technical support (36 percent).'’? The latter 
suggests that respondents may draw a con- 
nection between the use of data produced 
by certain organizations with the financial or 
technical support these organizations pro- 
vide to them or their government.”° Leaders 
viewed domestic data as helpful when it was 
available at the right level of aggregation, 
as well as being timely, trustworthy, and in- 
sightful. 


In the 2017 Education Snap Poll, we went a 
step further to ask education decision-mak- 
ers the three most important improvements 
that producers could undertake to improve 


19 The percentage of respondents in the health and 
governance sector that indicated that information from 
international sources was helpful because it was accompa- 
nied by financial, material or technical support was lower 

8 percent and 25 percent, respectively). Instead the third 
most important reason according to health (governance) 
respondents was that information was timely and up-to-date 
unbiased and trustworthy). 


20 For instance, respondents using the World Bank's data 

o improve their performance on certain development 
indicators may view this as a way to signal their commitment 
o reforms and thus be more likely to receive financial or 
echnical assistance from the Bank. 


their data. Overall, survey responses suggest 
that improving the timeliness and accessibil- 
ity of available data matter most to end users 
(Figure 13).71 Over half of the decision-makers 
surveyed also identified efforts to improve 
data disaggregation, accuracy, and trustwor- 
thiness as desirable. 


Despite broader transparency commit- 
ments” in many low- and middle-income 
countries, much government data is still pro- 
prietary or hidden behind paywalls (Custer 
and Sethi, 2017). Survey respondents would 
like data from the national government, 
in particular, to be more accessible and 
disaggregated (Figure A7, Appendix A). The 
dual emphasis here on accessibility of more 
granular data may indicate an untapped op- 
portunity: while reported use of subnational 
data lags behind national-level data, this 
may reflect a dearth of disaggregated infor- 
mation, rather than muted interest. 


Interestingly, education decision-makers 
also indicated that it would be helpful for 
education and policy experts to improve the 
accessibility of their data. This may reflect 
the need for these non-governmental actors 
to ramp up their own transparency efforts, 
as well as simplify documentation of their 
methodology and findings to be more easily 
understood by audiences with relatively less 
technical skill. 


When it comes to data produced by local 
governments, organizations, and schools, 
decision-makers place greater emphasis 
on improving timeliness. This response 
perhaps reflects the expected use for local 
information as compared with information 


21 The snap poll respondents were only asked to suggest 
improvements to sources of information that they deemed 
as being helpful in their work. 

22 The lack of Freedom of Information Laws is an additional 
constraint in many countries, though its existence does not 
necessarily guarantee freely accessible information. 


23 Data from development partners additionally lacked 
comparability over time. 
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produced by the national government. Since 
national government data may be used 
more for long-range planning or analysis 
of historical trends and the effectiveness of 
past policies, perhaps they do not need to be 
as up-to-date as data used for, say, program 
decisions in a local area. 


Government-produced data is sometimes 
characterized by a trust deficit (Custer and 
Sethi, 2017). While this information is in 
high demand, users do not see it as entirely 
credible when it is available. Several factors 
may be contributing to this request on the 
part of decision-makers to strengthen the 
trustworthiness of education data. Technical 
constraints including publication delays, 
episodic data collection, and limited data 
management capacity at the subnational 
level compound the problem of inaccurate 
data. Data at the point of service delivery 
(school or clinic-level data) is still largely 
paper-based, and digitization increases the 
risk of errors as well as delays. Political factors 
can also exacerbate inaccuracies in data, if 
incentives exist for public servants to mas- 
sage official numbers. 


Finding 6: Decision-makers strongly 
support strengthening their 
countries’ EMIS to bolster their 
education data ecosystem. 


Beyond identifying general areas of improve- 
ment for education data, respondents to the 
2017 Education Snap Poll also ranked a list 
of more specific solutions. Respondents were 
largely in agreement with the seven solutions 
proposed, rating all of them as “extremely 
important”, on average (see Figure 14). Of 
the seven solutions, the recommendation to 
strengthen the EMIS within the education 
ministry resonated with the highest number 
of respondents. 


In this respect, there is a nice symmetry 
between the call to strengthen the coun- 
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try-owned EMIS that arose from the survey 
of the literature in Chapter 2 and the favored 
solution of education decision-makers we 
analyzed in Chapter 3. To realize this solution 
in practice, domestic leaders and their devel- 
opment partners will need to address several 
common EMIS shortcomings we discussed 
at length earlier in this paper, namely: frag- 
mentation of data collection activities across 
ministries, unclear protocols for sharing and 
disseminating data openly, lack of funding, 
poor coordination among _ international 
funders, and a nascent or deficient statistical 
culture (UNESCO, 2016). 


In this chapter, we distilled insights on the 
state of data use from two AidData surveys 
of education stakeholders in low- and mid- 
dle-income countries: the 2017 Listening 
to Leaders Survey and the 2017 Education 
Snap Poll conducted in partnership with the 
Brookings Institution. Our three-fold aim for 
this analysis has been to: (1) understand the 
status quo of how education sector leaders 
use data or analysis in the context of their 
most common decisions; (2) pinpoint areas 
of opportunity for producers and funders 
of education data to close gaps in response 
to revealed demand from end users; and 
(3) identify the most important areas and 
promising solutions to increase the value of 
education data in future. Armed with these 
insights, we conclude in Chapter 4 with sev- 
eral implications for the future of education 
data investments. 


| Domestic International 


Reflects the local context 


Available at the right 38% 
level of aggregation 8% 


29% 
Timely and up-to-date 
22% 


27% 
Unbiased and trustworthy 
13% 


27% 
Provides new insights 
23% 


Provides specific 21% 


policy recommendations 43% 


Draws upon information being 19% 


generated by governments 16% 


; 17% 
Published frequently 
0% 


Contains info. that government 14% 
officials cared about 13% 


Transparent in methods 9% 
and assumptions 22% 


Accompanied by financial/ 8% 
material/technical rewards 36% 


2% 
Easy to adapt for a new purpose 
2% 


1% 
Easy to understand 
8% 


Used by other governments | 0% 
that we emulate 1% 


Notes: The figure reports the percentage of respondents who cited each factor as a reason for why they rated certain information sources as 

particularly helpful. This figure is based on 32 (or 34) respondents who answered a question on what makes information from a given domestic (or 
international) organization particularly helpful. Respondents could select up to three reasons. 

Source: 2017 Listening to Leaders Survey. 30 


More timely 


More accessible 


More disaggregated 


More accurate 


More trustworthy 


More comparable over time 


Easier to understand & use 


No improvement 
needed 5% 


54% 


54% 


53% 


49% 


45% 


PROPORTION OF RESPONDENTS THAT SELECTED RESPONSE AT LEAST ONCE 


Notes: Respondents could select up to three improvements for each data source. 


Source: 2017 Education Snap Poll. 


‘Strengthen the Education Management and 
Information System within the Ministry of 
Education’ 


"Stakeholders at various levels in the education 
system should use data and analysis’ 


‘External education funders should require 
that all countries maintain good-quality and 
timely education data’ 


‘Education spending should be monitored 
and evaluated for impact’ 


‘Collect more data at the school and 
student levels’ 


‘Citizens and parents should demand more 
open data at the school and student levels’ 


‘All education data and evidence should be 
publicly accessible, or any required permission 
to use data should be easy to obtain’ 


Extremely Very Moderately Slightly Not at all 


Notes: All respondents were presented with the same list of possible solutions and could rank each as “extremely important’, “very important”, “moderately 


important’, “slightly important”, or “not at all important’. 
Source: 2017 Education Snap Poll. 


4.Conclusion and 


key findings 


Developing countries face multiple challeng- 
es in improving their education system’s abil- 
ity to meet ambitious goals related to access, 
quality, and equity. Limited resources, as well 
as poor or missing information on various di- 
mensions of the system, hampers progress. 
Increasing the availability and use of data 
and evidence is a critical arena for leadership 
and management in the education sector. 
The call for more and better data has been 
heard—and while investment in education 
data lags behind some other sectors, it has 
increased and improved substantially. How- 
ever, a data-driven system is not just about 
generating data. It is also about increased 
appreciation for and use of evidence. 


Several factors impede data use. Deci- 
sion-makers and other stakeholders may not 
know what data are available if data produc- 
ers do not invest enough in dissemination or 
sharing the data widely, if at all. The available 
data may not be the right evidence, that is, 
evidence that is relevant to the decisions 
and issues at hand, available at the time it is 
needed, and in a form that can be accessed, 
understood, and applied. 


This report has focused on the use of ev- 
idence—by whom, for what specific roles 
and decisions, and what type and sources of 
evidence. Political interests and low imple- 
mentation capacity that undermine the will- 
ingness and ability of actors to use evidence 
may block the pathway from data generation 
to use and to data-informed decisions. We 
have reviewed research about the effect of 


these factors to explain how best to increase 
evidence use. We have also presented data 
from two surveys that provide information 
on evidence use by and data needs of ed- 
ucation stakeholders in low- and middle-in- 
come countries. 


To conclude, we summarize the key findings 
on how an education system can transform 
itself from being data-rich to data-driven, 
and thus better able to meet its goal of learn- 
ing for all. Being able to increase evidence 
use, in turn, alters the benefit-cost calculus 
for data generation, thus creating a stronger 
information chain. 


Investments in an effective national 
education information system will 
pay off in terms of better data and 
more data use 


Shortcomings in the quality of education 
data—inaccuracies, uneven coverage, closed 
data, and delays in availability—erode trust in 
information and eventually discourage use. 
But perhaps more important than empha- 
sizing the improvements needed in specific 
data is developing a functional, reliable in- 
formation system that not only collects and 
stores data but also has the capacity to share 
and disseminate data to inform views and 
decisions. In many countries, however, the 
EMIS is limited by weaknesses in its struc- 
ture, capacity, and implementation. Its data 
sources tend to be fragmented across dupli- 
cative information systems within the same 
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ministry, or worse, across several ministries in 
charge of different sub-sectors. These weak- 
nesses often lead to, as well as stem from, 
inadequate and unpredictable funding. 


A systematic national learning assessment 
is a clear asset in education systems that 
aim to improve student learning, but these 
assessment mechanisms are still nascent in 
most developing countries. National learning 
assessments of language and mathematics 
at early grades and at the end of primary ed- 
ucation were carried out in just over one-half 
of developing countries in 2010-2015, and 
the coverage of lower secondary education 
is Sparser across countries and regions. There 
remain stark differences in this respect be- 
tween low- and middle-income countries. 


Besides increasing the number of countries 
with national assessments, a big task at 
hand is to strengthen those that exist. As a 
case in point, notwithstanding shortcom- 
ings, Jordan’s education system illustrates 
a long-term commitment to using student 
assessments to drive significant curriculum 
and other reforms. When Jordan fell in the 
rankings of international student assess- 
ments, the government did not “shoot the 
messenger’; rather, it sought to improve its 
own assessment system so it could have 
a better way of measuring and tracking 
student learning, and it continues to bench- 
mark its students’ performance against other 
countries (Obeidat and Dawani, 2014). 


Data and analysis that are 
trustworthy, shared, and 
disseminated will be noticed and 
used 

Barriers to timely access and use of data, es- 
pecially at the local level, waste investments 
in data generation and may result in loss of 
budgetary support. Nascent EMIS do not 
have clearly established protocols for sharing 
data, so data remain closeted and grow old 
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without being noticed, analyzed, or used. 
Some blockages are also because political 
interests can capture data systems. Ensuring 
that data are open and trustworthy implies 
some level of administrative independence 
from political interests throughout the infor- 
mation cycle. Investments in review and veri- 
fication processes by a third party will pay off. 


Improvements in subnational data 
collection protocols will bring data 
closer to decision and action points 


Decision-makers report using national-level 
data most often, but also indicate a desire for 
more disaggregated or local data. They want 
this level of data particularly for information 
provided by the national government and 
development partners. It is important that 
sub-national data, often collected, reported 
and entered into digital systems by local 
government officials, are reliable and trusted 
by all decision-makers and stakeholders. This 
quality assurance can be achieved in two 
ways. First, governments and development 
partners should invest in building the capac- 
ity of local officials to collect better data and 
encourage data use. When data collectors 
become users, the quality of information 
improves. Second, government ministries at 
the national level need to collaborate with 
local governments to determine the essen- 
tial information needed for decision-making. 
This will prioritize the types of information 
expected from local governments and re- 
duce the burden of data collection. 


Program evaluation evidence is 
regarded as essential to education 
decision-makers—but there’s not 
enough of it 

Education decision-makers reported pro- 
gram and project evaluation data to be 
the most essential to their work, used most 
often, and most desired. This type of infor- 
mation seems to be highly valuable but in 


short supply. The gap may lie in two areas. 
First, impact evaluations in education are 
heavily concentrated in a few countries and 
a few topics. Second, gaps in coverage aside, 
policymakers’ ability to interpret the evi- 
dence and link it to policy decisions may be 
limited. To address this, evidence producers 
should invest in communicating the results 
of their research in ways that various stake- 
holders can easily understand. In the process 
of communicating research and evidence, 
producers should pay careful attention to 
highlighting the generalizability of findings 
and any caveats that decision-makers should 
know, as well as how the findings from the 
research can be useful in programmatic and 
policy decisions. Policymakers could benefit 
from being presented with a range of solu- 
tions relevant to a country’s context, demon- 
strating what has and has not worked in the 
past, and— importantly—why and why not. 
To the extent possible, researchers should 
involve policymakers in the design and ex- 
ecution of impact evaluations to increase 
their salience and relevance. Finally, to build 
trust in the evidence, research should draw 
lessons not only from successes but also 
from failed initiatives. 


Decision-makers want different 
types of information, depending on 
the nature of their work 


Decision-makers have a host of met and 
unmet data needs, and those data needs 
differ depending on the decision. For ex- 
ample, decision-makers who focus on plan- 
ning and structures say they would benefit 
from program evaluation data, while those 
whose primary work relates to instructional 
matters report wanting more student-level 
assessment data, and those who are re- 
sponsible for resource management want 
better government budget and expenditure 
data. These results highlight the importance 
of targeting data generation and dissem- 
ination to particular constituencies. While 


producers may invest in more broad-based 
dissemination strategies for a certain data 
types, they may lose out on power users if the 
decision-makers who need them most do 
not know about or have access to the data 
they need the most. 


In this report, we have focused on the issues 
related to one part of the information cycle— 
data and evidence use. The lack of data use 
is lamented often by data producers and 
researchers but is usually taken for granted. 
The focus of investments in the information 
cycle has been solidly on data generation 
and improvements in the coverage and 
quality of evidence. While these improve- 
ments should promote data use (as in a 
positive feedback loop), the uptake and use 
of information is not automatic or straight- 
forward. Understanding why education de- 
cision-makers and influencers do not notice, 
value, or use data that are produced by their 
own statistical agencies or by international 
organizations deserves more attention than 
it has received thus far. Funders of data 
collection should continue to allocate more 
resources to disseminating evidence and 
monitoring evidence use, but more research 
on the effects of behavioral and contextual 
determinants of evidence use is warranted. 
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Appendix A: Composition 
of survey participants and 
additional figures 


Composition of survey participants 


Survey 2017 Education Snap Poll 2017 Listening to Leaders Survey 
Stakeholder Group Full Sample of Members of the Full Sample of Sample of Education 
Respondents | Sampling Frame Respondents Sector Respondents 


Host government 76 (42.5%) 27,990 (47.9%) 1,473 (45.8%) 63 (40.4%) 
Development partner 75 (41.9%) 14,502 (24.8%) 516 (16.1%) 19 (12.2%) 
CSO/NGO 16 (8.9%) 7,063 (12.1%) 701 (21.8%) 45 (28.8%) 
Private sector 1 (0.6%) 1,949 (3.3%) 179 (5.6%) 6 (3.8%) 
Country Experts TI (6.2%) 6,881 (11.8%) 345 (10.7%) 23 (14.7%) 
Total 179 58,385 3,214 156 


Notes: The reported number of respondents for the 2017 Listening to Leaders Survey includes only those respondents who indicated working as part of 
one of the five stakeholder groups listed above. All those who indicated working for none of those groups (N=89) were excluded from our analysis. 


Survey 2017 Education Snap Poll 2017 Listening to Leaders Survey 
World Bank Region Full Sample of Members of the Full Sample of Sample of Education 
Classification Respondents | Sampling Frame Respondents Sector Respondents 


East Asia and Pacific 44 (24.6%) 8,713 (14.9%) 474 (14.8%) 26 (16.7%) 
Europe and Central Asia 34 (19.0%) 10,247 (17.6%) 674 (21.0%) 32 (20.5%) 
Latin America and the 6 5 6 5 

Caribbean 13 (7.3%) 8,010 (13.7%) 424 (13.2%) 19 (12.2%) 
Peete wun ap 14 (7.8%) 5,767 (9.9%) 251 (7.8%) 12 (7.7%) 
South Asia 19 (10.6%) 5,427 (9.3%) 34] (10.6%) 17 (10.9%) 
Sub-Saharan Africa 55 (30.7%) 20,221 (34.6%) 1,050 (32.7%) 50 (32.1%) 
Total 179 58,385 3,214 156 


Notes: Numbers in each column add to 100%. 
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Additional figures 


fl Involved (N=152) Most Important (N=112) 
Planning and developing strategies ade 
Other 
Budgeting for education resources 
Designing programs and content 
Designing student activities 
Testing or credentialing students 
Assessing teacher performance 


Ensuring provision of school inputs 


Hiring teachers or principals 


Determining compensation 
for teachers/principals 


Creating or abolishing schools/grades 


0% 


PERCENTAGE OF RESPONDENTS 


Notes: This figure is based on question 3 and 6 of the snap poll questionnaire. Each respondent could select as many activities as applicable.. Of the 27 
responses for “other” for the “Most important” question, 20 could be recoded to the four decision-domains. 
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Planning and structures 


Organization of instruction 


Resource management 


Other 


Personnel management 


PERCENTAGE OF RESPONDENTS 


52.7% 


Notes: This figure combines the ten activities in the snap poll into four decision-making domains, adapting from OECD (2012) 


Activities in the education sector 


Most important factor in making or shaping 


decisions 


Designing and defining programs of study and 
course content 


Designing and implementing support activities for 


students 

Creating or closing/abolishing schools or grades 
Testing, assessing, and/or credentialing students 
Hiring and deploying teachers and/or principals 


Developing careers and assessing performance of 
teachers and/or principals 


Determining compensation of teachers and/or 
principals 


Budgeting and allocating financial resources for 
education 


Ensuring provision of school inputs 


Planning and developing strategies 


Source: 2017 Education Snap Poll 


Government has sufficient capacity 


Sufficient financial resources; Government has 
sufficient capacity 


Enough information; Public and political support 
Government has sufficient capacity 
Government has sufficient capacity 


Public and political support 


Sufficient financial resources 


Public and political support 


Sufficient financial resources 


Government has sufficient capacity 
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Education administrative data 52% 


Program/project performance and 51% 


evaluation data and analysis 

Government budget and expenditure data 
Student-level assessment data 

Survey data 

School-level data 


Teacher performance data 


Citizen opinion data 


Aid and/or philanthropic finance data 


PERCENTAGE OF RESPONDENTS 


Notes: The figure is based on question 7 in the snap poll questionnaire and respondents could select up to three types of data. The figure shows the 
number of responses that selected each information source as among most essential types of information. Numbers in parentheses are the percentage of 
respondents. The figure excludes responses from two respondents that selected “other”. 

Source: 2017 Education Snap Poll 


Program/project performance and 


fe) 
evaluation data and analysis een 


School-level data 

Citizen opinion data 

Government budget and expenditure data 
Teacher performance data 

| have access to the information | need 
Education administrative data 
Student-level assessment data 

Survey data 


Aid and/or philanthropic finance data 


PERCENTAGE OF RESPONDENTS 


Notes: The figure is based on question 8 in the snap poll questionnaire and respondents could select up to three types of data. The figure shows the 
number of times each information type was selected by respondents. Numbers in parentheses are the percentage of respondents. The figure excludes 
responses from four respondents that selected “other”. 

Source: 2017 Education Snap Poll 
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Planning and 
structures [120] 


Organization of 
instruction [45] 


Resource 
management [32] 


Personnel 
management [16] 


ees __ eee 
=lLlUMU 
—— lo 
ews 


Education Student- School- Teacher Survey Citizen Program Government Aid/ | have the 
admin. level level performance data opinion performance/ — budget/_ philanthropic information 
data assessment data data data evaluation expenditure finance |need 
data data and data data 
analysis 


Notes: The figure reports Figure A4 by decision-domains, visualizing the distribution of responses within each decision-domain from left to right. Number in 
parentheses is the number of responses in each decision-domain. 
Source: 2017 Education Snap Poll 


N 
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Development 
Partner [98] 


Host 
Government [92] 


CSO/NGO [20] 


Independent 
Country 
Expert [15] 


Education Student- School Teacher Survey Citizen Program Government Aid/ | have the 
admin. level level performance data opinion performance/ budget/ philanthropic information 
data assessment data data data evaluation expenditure finance I need 
data data and data data 
analysis 


Notes: The figure reports Figure A4 by stakeholder group, visualizing the distribution of responses within each decision-domain from left to right. Number 
in parentheses is the number of responses for each stakeholder group. 
Source: 2017 Education Snap Poll 
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School-generated [72] 


Policy / education 
experts [92] 


News media [6] 


Citizen-generated [32] 


By local 
organizations [44] 


By local 
government [86] 


By government [140] 


By development 
partners [132] 


More Easier to More More More More More None 
timely understand trustworthy disaggregated accessible comparable accurate needed 
and use over time 


Notes: This figure is based on question 11 in the snap poll questionnaire, visualizing the distribution of responses for each information source from left to 
right. The number in parentheses is the number of responses for each information type. 
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Appendix B: Types of 
education data 


The foundation of an education informa- 
tion system is a school census. The specific 
indicators that are collected will reflect the 
goals of the education system, but a census 
typically captures school-level data on stu- 
dents, staffing, finances, infrastructure, and 
the administrative context. 


Countries can then aggregate raw school-lev- 
el data to gain valuable insight into the func- 
tioning of the education system in terms of 
access, quality, equity, and efficiency at all 
levels, particularly when paired with contex- 
tual information captured in household sur- 
veys and learning assessments that measure 
learning at a particular age or grade. 


Access indicators: Gross enrollment rate; 
net enrollment rate; school intake num- 
bers; average class size 


Quality indicators: Student/teacher ra- 
tios; classroom size; student assessment 
and performance data 


Efficiency indicators: Completion/repe- 
tition rates; dropout rates; transition rates; 
survival rates; cost-benefit data 


Equity indicators: Enrollment rates 
disaggregated by age group, gender, 
ethnicity, disability data, and other char- 
acteristics tied to the goals of the educa- 
tion system 


Student data School data Financial data 


Enrollment rates 
and non-teaching 


« Attendance and 
staff 


absenteeism rates 
* Teachers’ 


e Repetition and yee 
qualifications 


dropout rates 


« Assessment and test 


rates 
SCOles 


* Student 
characteristics (e.g, 
parents’ level of 
education and socio- 
economic standing; 
gender, ethnicity; 
language; disability) 
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Number of teaching 


Teachers’ attendance 


¢ School type (e.g., ¢ Source(s) of funding, 
public; private; 
religious) funding, tuition fees, 


° Facilities (e.g, grants, etc 


classrooms; e Allocation and 
electricity; bathrooms, Expenditures 
computers; furniture) 


e Resources (€.g., 
textbooks; paper) 


Sa government 


A high-quality Education Management 
Information System (EMIS) enables deci- 
sion-makers to draw on such various sources 
of information, also including labor market 
and living standards surveys, as well as 
census data, budgetary information, and 
data housed by ministries outside of the 
education sector. EMIS enables comparisons 
between schools, districts, and provinces/ 
states. Annual country data, such as those 
reported to UNESCO Institute for Statistics, 
are also used for cross-country comparisons 
and global benchmarking of basic outcome 
indicators. Table Bl shows how different 
international organizations, partnering with 
statistical and education agencies in coun- 
tries, have been collecting an array of educa- 
tion data, and have been making those data 
publicly available. 
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Appendix C: Available 
datasets on selected 
education indicators 


Dataset 


UIS Data 
Centre 


EdStats* 


Demographic 
and Health 
Survey (DHS) 


Multiple 
Indicator 
Cluster 
Surveys (MICS) 


Organization 


UNESCO 
Institute for 
Statistics 


World Bank 


ICF 
International. 
USAID 


UNICEF 


UN member 
countries 
responding 
to an annual 
survey 


UIS, WB, UN 


Household 
survey data 


Household 
survey data 
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Years 
available 


Annual, 
1970-2015 


Annual, 

1970-2015, 
projection 
until 2050 


Intermittent, 
usually every 
five years 


Intermittent, 
varies by 
country 


Variables covered 


Out-of-school children, 
entry, participation, 
progression, completion, 
literacy, educational 
attainment, international 
student mobility, human 
and financial resources, 


teaching conditions, adult 


education, education 
system, population 


Out-of-school children, 
participation, progression, 
completion, literacy, 
educational attainment, 
human and financial 
resources, learning 
outcomes, population; 
public expenditures, 
labor, EMIS, population 


Participation, literacy, 
educational attainment, 
individual and household 
characteristics 


Participation, progression, 
completion, literacy, 
educational attainment, 
individual and household 
characteristics 


Number 
of 
countries 


218 


242 


89 since 
1984 
(48% in 
Sub- 
Saharan 
Africa) 


109 in 
total 
since 
1995/6 


Number 


Dataset Organization Keak 7 Variables covered of 
available ‘ 
countries 

Living World Bank Household Intermittent, Participation, educational 37 since 

Standards survey data varies by attainment, expenditures, 1985 

Measurement country individual and household 

Surveys (LSMS) characteristics 

Eurostat European UNESCO-OECD- Annual, EU Entry, participation, 28 since 
Commission Eurostat and countries progression, completion, 1995 

other surveys international student 


mobility, human and 
financial resources, 
teaching conditions, 
educational attainment, 
languages, transition 
from education to 
work, education system, 


population 
Organization @EED UNESCO-OECD-. Annual, Entry, participation, 41 since 
of Economic Eurostat and @EED progression, completion, 1995 
Cooperation & other surveys countries international student 
Development mobility, human and 
(OECD) financial resources, 


teaching conditions, 
educational attainment, 
education system, 


population 
Other Different Household Intermittent, Participation, literacy, 
household providers survey data varies by educational attainment, 
surveys*™ country individual and household 
characteristics, other 
information 


Note: *EdStats contains education data provided by the UIS combined with data from other sources. **E.g., Malaysia Family Life Survey, Indonesia Family Life 
Survey from RAND, and STEP Surveys from the World Bank, among others. 
Source: Adapted from UIS (2016) 
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Appendix D: Survey 
methodology and 
questionnaires 


Details on the 
implementation of the 2017 
Listening to Leaders Survey 


Prior to fielding the 2017 Listening to Lead- 
ers Survey, our research team spent nearly 
two years preparing a sampling frame of 
approximately 58,000 host government and 
development partner officials, civil society 
leaders, private sector representatives, and 
independent experts from 126 low- and low- 
er-middle income countries and semi-au- 
tonomous territories. In this appendix, we 
provide an overview of our methodology 
and describe key attributes of our sampling 
frame construction, questionnaire design, 
survey implementation, and data aggrega- 
tion processes. 


DEFINING THE POPULATION OF INTEREST 
While the true global population of devel- 
opment policymakers and practitioners is 
for all intents and purposes unobservable, 
we took painstaking efforts to identify a 
well-defined and observable population of 
interest. We define this population of inter- 
est as including those individuals who are 
knowledgeable about the formulation and 
implementation of government policies and 
programs in low- and lower-middle income 
countries at any point between 2010 and 
2015. For more information on sampling 
frame inclusion criteria, see Appendix C of 
Masaki et al. (2017). 
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In recognition of the need for cross-country 
comparability, and the fact that every gov- 
ernment consists of a unique set of institu- 
tions and leadership positions, we identified 
our population of interest by first mapping 
country-specific public sector institutions 
(and leadership positions within those insti- 
tutions) back to an ideal-typical developing 
country government. This ideal-typical gov- 
ernment consisted of 33 institution types, 
such as a Ministry of Finance, a Supreme 
Audit Institution, and a National Statistical 
Office. We then identified functionally equiv- 
alent leadership positions within these insti- 
tutions, and the specific individuals who held 
these positions between 2010 and 2015. For 
the four additional stakeholder groups that 
we included in our sampling frame (in-coun- 
try development partners, domestic civil so- 
ciety and non-governmental organizations, 
private sector associations, and independent 
experts), we undertook a similar process of 
first mapping country-specific institutions 
and positions, and then identifying the in- 
dividuals who held those positions between 
2010 and 2015. 


Identifying functional equivalents at the 
institution- and leadership position-level 
resulted in a sampling frame that enables 
comparison across countries. In addition, 
by clearly defining a population of interest 
and constructing a master sampling frame 
that is stratified by country, stakeholder 
group, and institution type, we managed to 
overcome one of the most vexing challenges 


associated with expert panels and opinion 
leader surveys: the absence of detailed de- 
mographic data and the inability to assess 
the representativeness of findings at various 
levels. The stratification of our master sam- 
pling frame by country, stakeholder group, 
and institution type makes it possible to 
generate extremely granular elite survey 
data that can be published at varying levels 
of disaggregation without compromising 
participant confidentiality. It also enables 
analysis of the factors that influence partici- 
pation rates as well as the underlying sources 
of response bias. A more detailed description 
of the master sampling frame can be found 
in Appendix C of Masaki et al. (2017). 


CREATING THE SAMPLING FRAME 

Our ability to select individuals from the 
population of interest for inclusion in our 
final sampling frame was constrained by the 
availability of individual contact information. 
We identified the contact information of 
potential survey participants using publicly 
available resources, such as organizational 
websites and directories, international con- 
ference records, Who's Who International, 
and public profiles on LinkedIn, Facebook, 
and Twitter. While we identified approx- 
imately 58,000 individuals who met our 
inclusion criteria in the sampling frame, we 
were able to identify and successfully send a 
survey invitation to roughly 47,000 of those 
individuals (about 80 percent). 


SURVEY IMPLEMENTATION 

We administered the 2017 Listening to 
Leaders Survey between early January and 
late March 2017. Survey implementation 
was guided by the Weisberg total survey 
error approach and the Dillman tailored 
design method. Survey recipients were sent 
a tailored email invitation to participate in 
the survey that included a unique link to 
the online questionnaire. During the course 
of the survey administration period, survey 
recipients received up to three different 
automated electronic reminders, as well as 


some additional tailored reminders. Survey 
participants were able to take the survey 
in one of six different languages: English, 
French, Spanish, Portuguese, Russian, and 
Arabic. Of 47,000 individuals who received 
our email invitation, about 3,500 indeed 
participated (with a response rate of 7.4%) 
and 1,441 survey respondents (41 percent) 
completed the survey. 


WEIGHTING SCHEME FOR AGGREGATE 
STATISTICS 

The response rate to the 2017 Listening to 
Leaders Survey was 7.4 percent. In light of this 
relatively low response rate and imperfect 
information about the representativeness 
of our sample vis-a-vis the sampling frame 
(ie. the population of interest), we employ 
non-response weights to account for unit 
non-response (or survey non-response) and 
to redress potential bias deriving from it. To 
generate non-response weights, we take the 
following steps. First, we estimate the prob- 
ability of survey response by using a logistic 
regression. For all members of our sampling 
frame, we have information on their gender, 
country, institution types (e.g., finance min- 
istry, anti-corruption agency, supreme audit 
institution) and stakeholder group (e.g., host 
government officials, develooment partners. 
We use all these predictors to estimate 
the probability of survey response for each 
member of the sampling frame (as each of 
them turns out to be significant in predict- 
ing survey response). Second, we take the 
inverse of the estimated probability to arrive 
at the final non-response weights used for 
our analysis. 
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The 2017 Listening to Leaders Survey 


Please note: This is not the complete 2017 Listening to Leaders Survey questionnaire. For brevity, we have 
presented only a subset of questions that were analyzed in this report. For the full survey questionnaire, 
please see the Appendix E of Masaki et al. (2017). 


Q]1 [SGI-4]': You've been selected to participate in this survey based on our records, which indicate 
that you worked [[in.country]] at some point between 2010 and 2015. In the drop-down menu below, 
please select the country you worked in for the longest period from 2010 to 2015. 


O <<List of 126 low-income and middle-income countries and semi-autonomous territories>> 
O | did not work in one of these countries at any point between 2010 and 2015. 


Q2 [SG1-4]: Please select the type of organization within [[Q1: Country]] with which you worked for the 
longest period of time between 2010 and 2015. 


O Government (1) 
O Development Partner (2) 

O Non-Governmental Organization or Civil Society Organization (3) 

O Private Sector (4) 

O | did not work for one of these types of organizations between 2010 and 2015. (5) 

Q2 [SC5]: Over your entire career, for approximately how many years have you monitored issues relat- 
ed to policy formulation and implementation in [[Q]: country]? 


O O-4 years (1) 
O 5-9 years (2) 

O 10-14 years (3) 

O 15-19 years (4) 

O More than 20 years (5) 

Q3 [SG1-4]: Please write the name of the organization within [[Q]: Country]] with which you worked for 
the longest period of time between 2010 and 2015.(Almost all of the questions in this survey will ask 
about your time at this organization.) 


Q3 [SG5]: We want to best capture your perspective on policy-making in [[Q]: Country]]. Starting as 
early as 2010, with which one of the following administrations are you most familiar? 


<<List of all administrations relevant for [[Q]l: Country]] between 2010 and 2015: not shown here to save 
space>> 


=" 


Some questions are asked to only a subset of respondents depending on where they worked. Our sampling frame consists of individuals who 
belonged to one of the five different stakeholder groups: host government (SG1); development partner officials (SG2); civil society leaders (SG3); 
private sector representatives (SG4); and independent experts (SG5). Indicated in brackets are stakeholder groups to which a given question 
was asked. 
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Q4 [SG1-4]: The following questions refer to your time at [[Q3: Organization]]. Please identify the posi- 
tion that you held for the longest period of time between 2010 and 2015. What was the name of this 
position? (example: Director) 


Q5 [SG1-4]: In which of the following years did you hold this position? 
Oo 2010 (1) 


ooooaa 
N 
S) 
W 


Q6 [SGI1-4]: The following question asks about your area of focus while holding the position of [[Q4: 
Position]]. What was your primary area of focus? (Please select one.) 


Q6 [SC5]: Thinking of [[Q3: Administration]], with which area of policy-making are you most familiar? 
(Please select one.) 


oO 


Agriculture, Fishing and Forestry (1) 

Economic Policy (2) 

Education (3) 

Energy and Mining (4) 

Environment and Natural Resource Management (5) 
Finance (6) 

Health (7) 

Human Development and Gender (8) 
Industry, Trade and Services (9) 

Information and Communications Technologies (10) 
Labor Market Policy and Programs (11) 
Nutrition and Food Security (12) 

Private Sector Development (13) 

Good Governance and Rule of Law (14) 

Public Sector Management (15) 

Rural Development (16) 

Social Development and Protection (17) 
Trade (18) 

Transportation (19) 

Urban Development (20) 

Water, Sewerage and Waste Management (21) 
Foreign Policy (22) 

Other (Please indicate): (23) 


QDoodooonoonnonaonoOonaonolUmonemCUm neon ooel ee ooo ol 


Q20 [SG1-4]: Now we would like to ask about the raw data and analysis you used while you were work- 
ing on [[Q6: Policy Area]] problems. 
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Q20 [SGI]: At which stages of the policy process have you used raw data in your work on [[Q6: Policy 
Areal] policy initiatives in [[Q]: Country]]? For the purposes of this survey, we define raw data as a data 
point, dataset, or datasets (examples: spreadsheet, CSV file). (Please select any and all that apply.) 


Oo 


Research and analysis (1) 

Advocacy and agenda-setting (2) 

Design (3) 

Implementation (4) 

Monitoring and evaluation (5) 

External communications (6) 

Training, capacity building, and/or technical support (7) 
Don't know/not sure (8) 

Prefer not to say (9) 

None of these (10) 


boodadaanauda 


Q20 [SG2-4]: For which purposes have you used raw data in your work on [[Q6: Policy Area]] policy ini- 
tiatives in [[Q]l: Country]]? For the purposes of this survey, we define raw data as a data point, dataset, 
or datasets (examples: soreadsheet, CSV file).(Please select any and all that apply.) 


Oo 


Research and analysis (1) 

Advocacy and agenda-setting (2) 

Design (3) 

Implementation (4) 

Monitoring and evaluation (5) 

External communications (6) 

Training, capacity building, and/or technical support (7) 
Don't know/not sure (8) 

Prefer not to say (9) 

None of these (10) 


Oo oa a: e of Be oe 


Q21 [SGI]: At which stages of the policy process have you used analysis in your work on [[Q6: Policy 
Areal] policy initiatives in [[Q]: Country]]? For the purposes of this survey, we define analysis as evalu- 
ations, papers, Memos, and other products that use interpretations of data to provide insight into a 
particular situation. (Please select any and all that apply.) 


Oo 


Research and analysis (1) 

Advocacy and agenda-setting (2) 

Design (3) 

Implementation (4) 

Monitoring and evaluation (5) 

External communications (6) 

Training, capacity building, and/or technical support (7) 
Don't know/not sure (8) 

Prefer not to say (9) 

None of these (10) 


bo of. oO Be b-a e 


Q2]1 [SG2-4]: For which purposes have you used analysis in your work on [[Q6: Policy Areal] policy 
initiatives in [[Q]: Country]]? For the purposes of this survey, we define analysis as evaluations, papers, 
memos, and other products that use interpretations of data to provide insight into a particular situa- 
tion. (Please select any and all that apply.) 
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Research and analysis (1) 

Advocacy and agenda-setting (2) 

Design (3) 

Implementation (4) 

Monitoring and evaluation (5) 

External communications (6) 

Training, capacity building, and/or technical support (7) 
Don't know/not sure (8) 

Prefer not to say (9) 

None of these (10) 


OboodaadnadOao odo 


Q27 [SG1-4]: Now we would like to ask about the raw data and analysis provided to your team by for- 
eign and international organizations while you were working on [[Q6: Policy Area]] problems. 


Q27 [SC5]: Now we would like to ask about the raw data and analysis you used to study and monitor 
[[Q6: Policy Area]] problems in [[Q1: Country]. 


Q27 [SG1-4]: In making decisions while working on [[Q6: Policy Area]] policy issues, did you use any raw 
data or analysis provided by foreign or international organizations? For the purposes of this survey, we 
define raw data as a data point, dataset, or datasets (examples: spreadsheet, CSV file) and analysis as 
evaluations, papers, memos, and other products that use interpretations of data to provide insight 
into a particular situation. 


O Yes, | used raw data or analysis provided by foreign or international organizations. (1) 
O No,|! did not use raw data or analysis provided by foreign or international organizations. (2) 


Q27 [SG5]: In your work studying and monitoring [[Q6: Policy Area]] policy issues in [[Q]: Country]], did 
you use any raw data or analysis? For the purposes of this survey, we define raw data as a data point, 
dataset, or datasets (e.g., a soreadsheet, CSV file) and analysis as evaluations, papers, memos, and oth- 
er products that use interpretations of data to provide insight into a particular situation. 


O Yes, | used raw data or analysis provided by foreign or international organizations. (1) 
O No,! did not use raw data or analysis provided by foreign or international organizations. (2) 


Q28 [SG1-4]: Which external sources of information have you drawn from? For the purposes of this 
survey, we define raw data as a data point, dataset, or datasets (examples: spreadsheet, CSV file) and 
analysis as evaluations, papers, memos, and other products that use interpretations of data to provide 
insight into a particular situation. 


<<List of foreign and international organizations being identified in Q22>> 


Q29 [SG1-4]: How did you become familiar with the information you used from external sources? For 
the purposes of this survey, we define familiar as being aware the information existed. (Please select 
any and all that apply.) 


<<List of foreign and international organizations being identified in Q28>> 


O Email/e-newsletters (1) 
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Informal face- to face communication (2) 
Memorandum/policy brief/short technical papers (3) 
Social media (4) 

Formal meeting or consultation (5) 

Internet search (6) 

Traditional media (newspaper, radio, television) (7) 
Information or data portal (8) 

Don't know / Not sure (9) 

None of these (10) 


oO 8 o-oo Bo 


Q30 [SGI1-4]: Which types of analyses, if any, did your team use from each of the following organiza- 
tions to support your work on this initiative? For the purposes of this survey, we define analysis as eval- 
uations, papers, Memos, and other products that use interpretations of data to provide insight into a 
particular situation. (Please select any and all that apply.) 


Q30 [SC5]: Which types of analyses, if any, did you use to support your work studying or monitoring 
this initiative? For the purposes of this survey, we define analysis as evaluations, papers, memos, and 
other products that use interpretations of data to provide insight into a particular situation. (Please 
select any and all that apply.) 


<<List of foreign and international organizations being identified in Q28>> 


O Qualitative analysis (1) 
O Quantitative analysis (2) 

O Impact evaluation analysis (3) 

O Another type of analysis (4) 

O Don't know / Not sure (5) 

Q31 [SG1-4]: Which types of raw data, if any, did your team use from each of the following organiza- 
tions to support your work on this initiative? For the purposes of this survey, we define raw data as a 
data point, dataset, or datasets (examples: spreadsheet, CSV file). (Please select any and all that apply.) 


Q31 [SC5]: Which types of raw data, if any, did you use to study and monitor this initiative? For the pur- 
poses of this survey, we define raw data as a data point, dataset, or datasets (examples: spreadsheet, 
CSV file). (Please select any and all that apply.) 


<<List of foreign and international organizations being identified in Q28>> 


Oo 


National statistics (1) 

Survey data (examples: household surveys, income surveys) (2) 
Public opinion data (3) 

Program/project performance and evaluation data (4) 
Government budget and expenditure data (5) 

Spatial or satellite data (6) 

Aid and/or philanthropic finance data (7) 

Another type of data (8) 

Don't know / Not sure (9) 


ob: 8 8 ob 8-8 
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Q32 [SG1-4]: What was the geographic scope of the information you used? (Please select any and all 
that apply.) 


Q32 [SG5]: What was the geographic scope of the information you used? (Please select any and all 
that apply.) 


<<List of foreign and international organizations being identified in Q28>> 


O Cross-national (1) 
National (2) 
Province/region (3) 
District (4) 

Village / town / city (5) 
Exact location (6) 

Don't know / Not sure (7) 
No data were featured (8) 


Oooadaaa ao 


Q33 [SG1-4]: You indicated that the foreign and international organizations below provided you with 
information. Overall, how helpful would you say the information provided by each of these foreign 
and international organizations was to your work? For the purposes of this survey, we define helpful as 
being of assistance in implementing policy changes. 


<<List of foreign and international organizations being identified in Q28>> 


Q34 [SG1-4]: You identified information from [[Q33: Organization]] as helpful. In your opinion, which 
type of information from [[Q33: Organization]] was most helpful? For the purposes of this survey, we 
define helpful as being of assistance in implementing policy changes. 


Qualitative analysis (1) 

Quantitative analysis (2) 

Impact evaluation analysis (3) 

Another type of analysis (4) 

National statistics (5) 

Survey data (examples: household surveys, income surveys) (6) 
Public opinion data (7) 

Program/project performance and evaluation data (8) 
Government budget and expenditure data (9) 

Spatial or satellite data (10) 

Aid and/or philanthropic finance data (11) 

Another type of data (12) 


Oboodoaoaoadoaodeaodocmvunondmnnemco mo 


Q35 [SGI-4]: What has made [[Q34: Information]] from [[Q33: Organization]] helpful? For the purposes 
of this survey, we define helpful as being of assistance in implementing policy changes. (Please check 
up to 3 boxes.) 


O It was easy to understand. (1) 
Oo It was easy to adapt for a new purpose. (2) 
O It contained information that senior government officials cared about. (3) 
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It provided new insights that were not otherwise understood or appreciated. (4) 
It reflected an understanding of the local context [[Q]: Country]]. (5) 

It was timely and up-to-date. (6) 

It provided a concrete set of policy recommendations. (7) 

It was used by other governments that we could emulate. (8) 

It drew upon data or analysis produced by the government. (9) 

It was based on a transparent set of methods and assumptions. (10) 

It was seen as unbiased and trustworthy. (11) 

It was accompanied by critical financial, material, or technical support. (12) 

It was published frequently. (13) 

It was at the right level of aggregation (i.e., cross-national, national, district) (14) 
Another reason (Please describe): (15) 

Don't know / Not sure (16) 

None of these (17) 

Prefer not to say (18) 


Oo. O° 6 bo oe. ae ae osB 


Q40 [SGlor5]: Please list the names of as many domestic organizations in [[Q]: Country] that provid- 
ed the government with advice or assistance to support this initiative as you can remember. (Please 
select as many as apply and/or write the full name of each organization. Do not include your own 
organization.) 


Q47 [SG1-4]: Now we would like to ask about the raw data and analysis provided to your team by do- 
mestic organizations while you were working on [[Q6: Policy Area]] policy problems. 


Q47 [SG1-4]: In making decisions while working on [[Q6: Policy Area]] policy issues, did you use any raw 
data or analysis provided by domestic organizations? For the purposes of this survey, we define raw 
data as a data point, dataset, or datasets (examples: soreadsheet, CSV file) and analysis as evaluations, 
papers, memos, and other products that use interpretations of data to provide insight into a particu- 
lar situation. 


O Yes, | used raw data or analysis provided by domestic organizations. (1) 
O No, |! did not use raw data or analysis provided by domestic organizations. (2) 


Q48 [SGI1-4]: Which domestic sources of information have you drawn from? For the purposes of this 
survey, we define raw data as a data point, dataset, or datasets (examples: soreadsheet, CSV file) and 
analysis as evaluations, papers, memos, and other products that use interpretations of data to provide 
insight into a particular situation. 


<<List of domestic organizations being identified in Q40>> 


Q49 [SG1-4]: How did you become familiar with the information you used from external sources? For 
the purposes of this survey, we define familiar as being aware the information existed. (Please select 
any and all that apply.) 


<<List of domestic organizations being identified in Q48>> 


O Email/e-newsletters (1) 
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Informal face- to face communication (2) 
Memorandum/policy brief/short technical papers (3) 
Social media (4) 

Formal meeting or consultation (5) 

Internet search (6) 

Traditional media (newspaper, radio, television) (7) 
Information or data portal (8) 

Don't know / Not sure (9) 

None of these (10) 


Ooooadaaadnaada 


Q50 [SG1-4]: Which types of analyses, if any, did your team use from each of the following organiza- 
tions to support your work on this initiative? For the purposes of this survey, we define analysis as eval- 
uations, papers, Memos, and other products that use interpretations of data to provide insight into a 
particular situation. (Please select any and all that apply.) 


<<List of domestic organizations being identified in Q48>> 


O Qualitative analysis (1) 
Quantitative analysis (2) 
Impact evaluation analysis (3) 
Another type of analysis (4) 
Don't know / Not sure (5) 


oooda 


Q51 [SG1-4]: Which types of raw data, if any, did your team use from each of the following organiza- 
tions to support your work on this initiative? For the purposes of this survey, we define raw data as a 
data point, dataset, or datasets (examples: spreadsheet, CSV file). (Please select any and all that apply.) 


<<List of domestic organizations being identified in Q48>> 


Oo 


National statistics (1) 

Survey data (examples: household surveys, income surveys) (2) 
Public opinion data (3) 

Program/project performance and evaluation data (4) 
Government budget and expenditure data (5) 

Spatial or satellite data (6) 

Aid and/or philanthropic finance data (7) 

Another type of data (8) 

Don't know / Not sure (9) 


Ooodaaandn od 


Q52 [SG1-4]: What was the geographic scope of the information you used? (Please select any and all 
that apply.) 


<<List of domestic organizations being identified in Q48>> 


O Cross-national (1) 
O National (2) 

O Province/region (3) 
O District (4) 
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Village / town / city (5) 
Exact location (6) 

Don’t know / Not sure (7) 
No data were featured (8) 


Ob) 8 


Q53 [SG1-4]: You indicated that the domestic organizations below provided you with information. 
Overall, how helpful would you say the information provided by each of these domestic organizations 
was to your work? For the purposes of this survey, we define helpful as being of assistance in imple- 
menting policy changes. 


<<List of domestic organizations being identified in Q48>> 


Q54 [SG1-4]: You identified information from [[Q53: Domestic Organization]] as helpful. In your opin- 
ion, which type of information from [[Q53: Domestic Organization]] was most helpful? For the purpos- 
es of this survey, we define helpful as being of assistance in implementing policy changes. 


Oo 


Qualitative analysis (1) 

Quantitative analysis (2) 

Impact evaluation analysis (3) 

Another type of analysis (4) 

National statistics (5) 

Survey data (examples: household surveys, income surveys) (6) 
Public opinion data (7) 

Program/project performance and evaluation data (8) 
Government budget and expenditure data (9) 

Spatial or satellite data (10) 

Aid and/or philanthropic finance data (11) 

Another type of data (12) 


bo 8-8 oa. ob 8 fb oe 


Q55 [SG1-4]: What has made [[Q54: Information]] from [[Q53: Domestic Organization]] particularly 
helpful? For the purposes of this survey, we define helpful as being of assistance in implementing pol- 
icy changes. (Please check up to 3 boxes.) 


oO 


It was easy to understand. (1) 

It was easy to adapt for a new purpose. (2) 

It contained information that senior government officials cared about. (3) 

It provided new insights that were not otherwise understood or appreciated. (4) 
It reflected an understanding of the local context in [[Q]1: Country]]. (5) 

It was timely and up-to-date. (6) 

It provided a concrete set of policy recommendations. (7) 

It was used by other governments that we could emulate. (8) 

It drew upon data or analysis produced by the government. (9) 

It was based on a transparent set of methods and assumptions. (10) 

It was seen as unbiased and trustworthy. (11) 

It was accompanied by critical financial, material, or technical support. (12) 

It was published frequently. (13) 

It was at the right level of aggregation (i.e., cross-national, national, district) (14) 
Another reason (Please describe): (15) 
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O Don't know / Not sure (16) 
O None of these (17) 
O Prefer not to say (18) 


Q56 [SG1-4]: How did your team use the [[Q54: Information]] provided by [[Q53: Domestic Organiza- 
tion]]? (You may select up to three statements.) 


O To better understand the [[Q6: Policy Area]] policy problems that needed to be solved (1) 
O To keep citizens and other domestic stakeholders updated on the initiative’s progress (2) 
O To keep foreign and international stakeholders updated on the initiative’s progress (3) 

O To advocate for the adoption or implementation of the initiative (4) 

O To make budgetary or resource allocation decisions (5) 

O To identify the [[Q6: Policy Area]] policy problems that were most critical to solve (6) 

O To design or inform specific implementation strategies (7) 

O To foster a broader partnership with [[Q53: Domestic Organization]] (8) 

O To monitor progress made towards solving specific [[Q6: Policy Area]] policy problems (9) 
O To petition for resources from authorizing entities or external partners (10) 

O To make course corrections during the implementation of the initiative (11) 

O Another reason (Please describe): (12) 

O Don't know / Not sure (13) 

O None of these (14) 

O Prefer not to say (15) 


Q57 [SG1-4]: What did this information help your team to accomplish? 


Q58 [SG1-4]: You identified that information from [[Q53: Domestic Organization]] could have been 
more helpful. What were the biggest challenges your team faced when trying to use information pro- 
vided by [[Q53: Domestic Organization]]? For the purposes of this survey, we define helpful as being of 
assistance in implementing policy changes. (Please check up to 3 boxes.) 


O It was hard to understand (1) 

O It was hard to adapt for a new purpose (2) 

O It did not contain enough information that government officials cared about (3) 
O It did not provide any new insights (4) 

O It did not reflect an understanding of the local context [[Q]: Country]] (5) 

O It was untimely and out-of-date (6) 

O It did not provide a concrete set of policy recommendations (7) 

O It had not been used by other governments that we could emulate (8) 

O |t did not draw upon data or analysis produced by the government (9) 

O It was not transparent in its methods or assumptions (10) 

O It was seen as biased and untrustworthy (11) 

O It was not accompanied by critical financial, material, or technical support (12) 

O It was received at a time when there was not much opportunity for change (13) 
O It was not specific enough (for example, with respect to stakeholder group or geography) (14) 
O Another reason (Please describe): (15) 

O Don't know / Not sure (16) 

O None of these (17) 

O Prefer not to say (18) 
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The 2017 Education Snap Poll 


Q1 Thank you for your interest in our snap poll, which should take you about 10 minutes to complete. 
We are inviting you to participate in this study because we believe you are knowledgeable about the 
education sector and education data in [[Q2: Country]] between 2010 and 2015. Your participation will 
help us ensure that our findings accurately represent the diverse opinions and observations of those 
with experience in education. Researchers from the College of William and Mary will use the results 
from this survey to undertake independent analysis of how education data is being used, and how it 
could be improved. 


This questionnaire is unique. We are seeking your individual observations and opinions based upon 
your own experiences. We are not seeking the official positions of any institutions with which you have 
been affiliated. Please complete as many questions as you can and feel free to express your views 
openly and honestly. If you do not feel comfortable answering a particular question for any reason, 
you are welcome to select the “don’t know” response and move on. 


Participation in this survey is completely voluntary. All of your responses will be kept strictly confiden- 
tial. They will only be used in a statistical summary and will never be associated with your name. We 
do not believe that there are any risks to participating in this survey. Every individual who chooses to 
participate in the survey will be granted access to our findings. 


If you have questions or need assistance in any way, please send an email to our research team (brad. 
parks@aiddata.org). Any ethical concerns with the conduct of the study should be directed to Thom- 
as Ward, Ph.D., Chair of the Protection of Human Subjects Committee at the College of William and 
Mary, at t)jward@wm.edu 757-221-2358. lf you would like to learn more about AidData’s survey work, 
please visit our website at www.aiddata.org/listening-to-leaders. 


Q2 In what country do you currently work? If your work is focused on more than one country, please 
select your country of greatest focus. 


Oo 


Afghanistan 
Albania 
Algeria 
Angola 
Armenia 
Azerbaijan 
Bangladesh 
Belarus 
Belize 
Benin 
Bhutan 
Bolivia 
Bosnia and Herzegovina 
Botswana 
Brazil 
Bulgaria 


ot ee CF Vt 6 VY 
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Oooo bo op oo oOo ooo opD oa ob oboe eb eo ob ep op ole eae ebb ee eb oe oo oo eo ee 


Burkina Faso 
Burundi 
Cambodia 
Cameroon 
Cape Verde 
Central African Republic 
Chad 

China 
Colombia 
Comoros 
DRC 

Congo 

Céte D'Ivoire 
Cuba 
Djibouti 
Dominican Republic 
Ecuador 
Egypt 

El Salvador 
Equatorial Guinea 
Eritrea 
Ethiopia 

Fiji 

Gambia 
Georgia 
Ghana 
Guatemala 
Guinea 
Guinea-Bissau 
Guyana 

Haiti 
Honduras 
India 
Indonesia 
lran 

Iraq 

Jamaica 
Jordan 
Kazakhstan 
Kenya 
Kiribati 
North Korea 
Kosovo 
Kurdistan 
Kyrgyzstan 
Laos 
Lesotho 
Liberia 


ee Oe OD OOO OO OD OOD OD OO 


Macedonia 
Madagascar 
Malawi 
Maldives 

Mali 

Marshall Islands 
Mauritania 
Federated States of Micronesia 
Moldova 
Mongolia 
Montenegro 
Morocco 
Mozambique 
Myanmar 
Namibia 

Nepal 
Nicaragua 
Niger 

Nigeria 
Pakistan 
Palestine 
Papua New Guinea 
Paraguay 

Peru 
Philippines 
Puntland 
Romania 
Rwanda 

Samoa 

Sao Tome and Principe 
Senegal 

Serbia 

Sierra Leone 
Solomon Islands 
Somalia 
Somaliland 
South Africa 
South Sudan 
Sri Lanka 
Sudan 
Suriname 
Swaziland 

Syria 

Tajikistan 
Tanzania 
Thailand 
Timor-Leste 
Togo 
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Tonga 

Tunisia 
Turkey 
Turkmenistan 
Tuvalu 
Uganda 
Ukraine 
Uzbekistan 
Vanuatu 
Vietnam 
Yemen 
Zambia 
Zanzibar 
Zimbabwe 

| do not work in or study one of these countries. 


Oboe fb oe eo oe 8 8 Be 


Q3 Which of the following education activities are you involved in? 
Select and any all that apply. 


O Designing and defining programs of study and course content 

O Designing and implementing support activities for students (e.g., after school remedial 
education) 

Creating or closing/abolishing schools or grades 

Testing, assessing, and/or credentialing students 

Hiring and deploying teachers and/or principals 

Developing careers and assessing performance of teachers and/or principals 
Determining compensation of teachers and/or principals 

Budgeting and allocating financial resources for education 

Ensuring provision of school inputs (e.g., school supplies, desks) 

Planning and developing strategies (e.g., writing a strategy, monitoring & evaluation) 
Other 

lam not involved in any of these activities 


OHoohaag aoa oa ob oe 


Q4 What is the nature of your involvement in those activities in [[Q3: Activities]? 


| make the final | provide support | advocate fora | play another 

decision related to policymakers course of action role related to 

to this activity. | who make deci- related to this this activity. 
sions related to activity. 


this activity. 
Designing and defining pro- oO oO oO oO 
grams of study and course 
content 
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Designing and implementing oO oO oO oO 
support activities for students 

(e.g., after school remedial 

education) 


Creating or closing/abolishing oO oO oO oO 
schools or grades 


Testing, assessing, and/or cre- oO oO oO oO 
dentialing students 


Hiring and deploying teachers oO oO oO oO 
and/or principals 


Developing careers and as- oO oO Ol Ol 
sessing performance of teach- 
ers and/or principals 


Determining compensation of oO oO oO oO 
teachers and/or principals 


Budgeting and allocating oO oO oO oO 
financial resources for educa- 


tion 


Ensuring provision of school oO oO oO oO 
inputs (e.g., school supplies, 
desks) 


Planning and developing oO oO oO oO 
strategies (e.g., writing a strat- 
egy, monitoring & evaluation) 


Other oO O O L] 
Other oO oO O LJ 
lam not involved in any of im O CO O 


these activities 
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Q5 What is the most important factor in making or influencing decisions within these activities? 


There is There are There is pub- The govern- Another 
enough in- | sufficient lic and politi- ment has factor 
formation financial cal support sufficient 
resources capacity 
Designing and defining programs oO oO oO Ol oO 
of study and course content 
Designing and implementing oO oO oO oO oO 


support activities for students (e.g., 
after school remedial education) 


Creating or closing/abolishing oO oO oO oO oO 
schools or grades 

Testing, assessing, and/or creden- oO oO oO oO oO 
tialing students 

Hiring and deploying teachers oO oO oO oO rl 
and/or principals 

Developing careers and assessing Ol oO oO oO oO 
performance of teachers and/or 

principals 

Determining compensation of oO oO oO oO oO 


teachers and/or principals 


Budgeting and allocating financial oO oO oO oO oO 
resources for education 


Ensuring provision of school inputs oO oO oO oO oO 
(e.g., school supplies, desks) 


Planning and developing strate- oO oO oO oO oO 
gies (e.g., writing a strategy, moni- 
toring & evaluation) 


Other oO oO O | L 
Other oO oO O | L 
lam not involved in any of these oO im ia ia oO 
activities 
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Q6 You previously indicated involvement in the activities listed below. Which is most important to 
your work? 


oO 


Oooodoaqdaaonoadvuauvdoo do 


Designing and defining programs of study and course content 

Designing and implementing support activities for students (e.g., after school remedial educa- 
tion) 

Creating or closing/abolishing schools or grades 

Testing, assessing, and/or credentialing students 

Hiring and deploying teachers and/or principals 

Developing careers and assessing performance of teachers and/or principals 
Determining compensation of teachers and/or principals 

Budgeting and allocating financial resources for education 

Ensuring provision of school inputs (e.g., school supplies, desks) 

Planning and developing strategies (e.g., writing a strategy, monitoring & evaluation) 
Other 

lam not involved in any of these activities 


Q7 Which of the following types of information are most essential to your work on [[Q6: Domain]]? 
Please select up to three. 


oO 


Oooododaduodvooomo mo 


Education administrative data (example: enrollment figures, number of schools) 
Student-level assessment data 

School-level data (example: pass/fail rates, share of minority students) 
Teacher performance data 

Survey data (examples: household surveys, income surveys) 

Citizen opinion data (example: survey data on citizen preferences) 
Program/project performance and evaluation data and analysis 
Government budget and expenditure data 

Aid and/or philanthropic finance data 

Other: 

| do not use any of these types of information in my work. 


Q8 Which of the following types of information do you wish existed for your work on [[Q6: Domain]]? 
Please select up to three. 


oO 


Oboodaoodcadmo oO do 


| have access to the types of information | need for my work. 
Education administrative data (example: enrollment figures, number of schools) 
Student-level assessment data 

School-level data (example: pass/fail rates, share of minority students) 
Teacher performance data 

Survey data (examples: household surveys, income surveys) 

Citizen opinion data (example: survey data on citizen preferences) 
Program/project performance and evaluation data and analysis 
Government budget and expenditure data 

Aid and/or philanthropic finance data 

Other: 
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Q9 Which of the following sources of information are most helpful to you in your work on [[Q6: Do- 
main]]? 
Please select up to three choices. 


oO 


Data provided by the Government of [[Q2: Country] 

Data provided by the government at the local level (district, province, municipality) 
Data provided by local organizations (example: non-governmental organizations) 

Data provided by development partners/donors (example: UNESCO, World Bank, DFID) 
Citizen-generated data (example: parent feedback on schools) 

School-generated data (example: student and teacher absences) 

Policy and/or education experts (e.g., think tanks, academics) 

News media 

Other 

| do not use any of these sources of information in my work 


Do ob ob 8 ob oo 


Q10 How do you usually access this information? 
Please select up to three options. 


oO 


Email/e-newsletters 


O Informal communication (example: face-to-face, phone calls) 

O Memorandum/policy brief/short technical papers 

O Social media 

O Formal meeting or consultation 

O Internet search (example: visiting websites related to education) 
O Traditional media (example: print newspaper, radio, television) 
O Online media (example: online newspaper, online magazine) 

O Information or data portal (example: EMIS) 

O Don't know/not sure 

Oo Other 


Qll What improvements would make information from these sources more helpful? 
You may select up to three improvements for each data source. 
Responses from [[Q8]] 


More Easier to More More More ac- More More Another The 
timely = under- _ trust- disaggre- cessible compa- accurate improve- source 
stand worthy gated rable over ment does not 
and use time need 
improve- 
ment 


Data provided by oO oO Ol | O O LJ LJ O 


the Government 
of [[Q2: Country]]} 
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Data provided by O Oo Oo oO oO oO O O O 


the government 
at the local level 
(district, province, 
municipality) 


Data provided by O Oo Oo oO oO oO O O | 


local organiza- 
tions (example: 
non-governmen- 
tal organizations) 


Data provided oO oO oO ‘a O O LJ CL] L 


by development 
partners/donors 
(example: UNES- 
CO, World Bank, 
DFID) 


Citizen-generated [] Oo oO oO | oO O | O 


data (example: 
parent feedback 
on schools) 


School-generat- oO O Oo oO oO oO Ol im O 


ed data (exam- 
ple: student and 
teacher absences) 


Policy and/or Oo O O O O O O O CO 


education experts 
(e.g., think tanks, 


academics) 
News media O oO oO O | | LJ O O 
Other O LJ LJ LJ LJ LJ LJ LJ LJ 


|donotuseanyof [ Oo oO oO O oO oO im | 


these sources of 
information in my 
work. 
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Q12 Thinking about the improvements you recommended above, how important is each of the possi- 
ble solutions below? 


portant 


Extremely im-Very import- Moderately — Slightly im- 


important portant 


Not at all 
important 


Strengthen the Education 
Management and Information 
System within the Ministry of 
Education 


O 


O 


O 


O 


O 


Collect more data at the 
school and student levels 


Stakeholders at various lev- 
els in the education system 
should use data and analysis 


All education data and ev- 
idence should be publicly 
accessible, or any required 
permission to use data should 
be easy to obtain 


Citizens and parents should 
demand more open data at 
the school and student levels 


Education spending should be 
monitored and evaluated for 
impact 


External education funders 
should require that all coun- 
tries maintain good-quality 
and timely education data 


Q13 Where do you work? (example: Ministry of Education) 
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Q14 What is your title? (example: Director of Primary Education Curriculum) 


Q15 What is the most advanced degree you have received? (example: BA in Communications) 


Q16 Where did you receive your highest degree? 


oO 


Afghanistan 
Albania 
Algeria 
Andorra 
Angola 
Antigua and Barbuda 
Argentina 
Armenia 
Australia 
Austria 
Azerbaijan 
Bahamas 
Bahrain 
Bangladesh 
Barbados 
Belarus 
Belgium 
Belize 

Benin 
Bhutan 
Bolivia 
Bosnia and Herzegovina 
Botswana 
Brazil 

Brunei Darussalam 
Bulgaria 
Burkina Faso 
Burundi 
Cambodia 
Cameroon 
Canada 
Cape Verde 
Chad 

Chile 

China 


[os ee ee DOOD OD 
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Bo 8 8 ob of oboe bo ob eb eb eo ee eo Beebe Bb ee eee soe 8B eo a oe ea a oe oe eB 


Colombia 
Congo 
Costa Rica 
Céte d'Ivoire 
Croatia 
Cuba 
Cyprus 
Czech Republic 
Democratic Republic of the Congo 
Denmark 
Djibouti 
Dominica 
Ecuador 
Egypt 

El Salvador 
Equatorial Guinea 
Eritrea 
Estonia 
Ethiopia 

Fiji 

Finland 
France 
Gabon 
Georgia 
Germany 
Ghana 
Greece 
Grenada 
Guatemala 
Guinea 
Guinea-Bissau 
Guyana 
Haiti 
Honduras 
Hungary 
Iceland 
India 
Indonesia 
Iran 

Iraq 

Ireland 
Israel 

Italy 
Jamaica 
Japan 
Jordan 
Kazakhstan 
Kenya 
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Boo oo oho bo eo eb ee ee eb ob eo eb ee ob eb ee eb ee eb 8b be eo ob oe oe op eo eo 2 


Kiribati 
Kosovo 
Kuwait 
Kyrgyzstan 
Laos 

Latvia 
Lebanon 
Lesotho 
Liberia 

Libya 
Liechtenstein 
Lithuania 
Luxembourg 
Macedonia 
Madagascar 
Malawi 
Malaysia 
Mali 

Malta 
Mauritania 
Mauritius 
Mexico 
Moldova 
Monaco 
Mongolia 
Montenegro 
Morocco 
Mozambique 
Myanmar 
Namibia 
Nauru 

Nepal 

New Zealand 
Nicaragua 
Niger 
Nigeria 
North Korea 
Norway 
Oman 
Pakistan 
Palau 
Palestine 
Panama 


Papua New Guinea 


Paraguay 
Peru 
Philippines 
Poland 


ee Oe Oe DO OO OD) 


Portugal 

Qatar 

Romania 

Russia 

Rwanda 

Saint Kitts and Nevis 
Saint Lucia 

Saint Vincent and the Grenadines 
Samoa 

San Marino 

Sao Tome and Principe 
Saudi Arabia 

Senegal 

Serbia 

Seychelles 

Sierra Leone 
Singapore 

Slovakia 

Slovenia 

Somalia 

South Africa 

South Korea 

South Sudan 

Spain 

Sri Lanka 

Sudan 

Suriname 

Swaziland 

Sweden 

Switzerland 

Syria 

Taiwan 

Tajikistan 

Tanzania 

Thailand 

The Central African Republic 
The Comoros 

The Dominican Republic 


The Federated States of Micronesia 


The Gambia 

The Maldives 

The Marshall Islands 
the Netherlands 
The Solomon Islands 
Timor-Leste 

Togo 

Tonga 

Trinidad and Tobago 


ee OO | 


Tunisia 

Turkey 
Turkmenistan 
Tuvalu 

Uganda 

Ukraine 

United Arab Emirates 
United Kingdom 
United States 
Uruguay 
Uzbekistan 
Vanuatu 

Vatican City 
Venezuela 
Vietnam 

Yemen 

Zambia 
Zimbabwe 
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Q17 In what year were you born? 


1937 
1938 
1939 
1940 
1941 

1942 
1943 
1944 
1945 
1946 
1947 
1948 
1949 
1950 
1951 

1952 
1953 
1954 
1955 
1956 
1957 
1958 
1959 
1960 
196] 

1962 
1963 
1964 
1965 
1966 
1967 
1968 
1969 
1970 
197] 

1972 
1973 
1974 
1975 
1976 
1977 
1978 
1979 
1980 
198] 

1982 
1983 


oO 


Boo ooo ooo eb ee eb eb eb oe eb obo ebook eo eb ob oe eb oe oo eb bob eb ee ee 
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1984 
1985 
1986 
1987 
1988 
1989 
1990 
199] 

1992 
1993 
1994 
1995 
1996 
1997 


os | 


Q18 Thank you for taking the time to complete this survey. Please click the next button to submit your 
answers. 


We will share the results with you when they are published later this year and your answers will 
remain confidential. They will never be associated with your name and will be reported as statistical 
summaries. If you have any questions, please don't hesitate to contact our research team at survey@ 
aiddata.wm.edu. 
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